add codemap script

Bumps [vite](https://github.com/vitejs/vite/tree/HEAD/packages/vite) from 6.2.2 to 6.2.3.
- [Release notes](https://github.com/vitejs/vite/releases)
- [Changelog](https://github.com/vitejs/vite/blob/v6.2.3/packages/vite/CHANGELOG.md)
- [Commits](https://github.com/vitejs/vite/commits/v6.2.3/packages/vite)

---
updated-dependencies:
- dependency-name: vite
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <support@github.com>

.devcontainer/Dockerfile

@@ -1,6 +1,6 @@
 # See here for image contents: https://github.com/microsoft/vscode-dev-containers/tree/v0.233.0/containers/python-3/.devcontainer/base.Dockerfile
 
-# [Choice] Python version (use -bullseye variants on local arm64/Apple Silicon): 3, 3.10, 3.9, 3.8, 3.7, 3.6, 3-bullseye, 3.10-bullseye, 3.9-bullseye, 3.8-bullseye, 3.7-bullseye, 3.6-bullseye, 3-buster, 3.10-buster, 3.9-buster, 3.8-buster, 3.7-buster, 3.6-buster
+# [Choice] Python version (use -bullseye variants on local arm64/Apple Silicon): 3, 3.10, 3-bullseye, 3.10-bullseye, 3-buster, 3.10-buster, etc.
 ARG VARIANT="3.10-bullseye"
 FROM mcr.microsoft.com/vscode/devcontainers/python:0-${VARIANT}
 

.devcontainer/devcontainer.json

@@ -6,7 +6,7 @@
 		"dockerfile": "Dockerfile",
 		"context": "..",
 		"args": { 
-			// Update 'VARIANT' to pick a Python version: 3, 3.10, 3.9, 3.8, 3.7, 3.6
+			// Update 'VARIANT' to pick a Python version: 3, 3.10, etc.
 			// Append -bullseye or -buster to pin to an OS version.
 			// Use -bullseye variants on local on arm64/Apple Silicon.
 			"VARIANT": "3.10",

.github/CODE_OF_CONDUCT.md

@@ -1,46 +0,0 @@
-# Contributor Covenant Code of Conduct
-
-## Our Pledge
-
-In the interest of fostering an open and welcoming environment, we as contributors and maintainers pledge to making participation in our project and our community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, gender identity and expression, level of experience, nationality, personal appearance, race, religion, or sexual identity and orientation.
-
-## Our Standards
-
-Examples of behavior that contributes to creating a positive environment include:
-
-* Using welcoming and inclusive language
-* Being respectful of differing viewpoints and experiences
-* Gracefully accepting constructive criticism
-* Focusing on what is best for the community
-* Showing empathy towards other community members
-
-Examples of unacceptable behavior by participants include:
-
-* The use of sexualized language or imagery and unwelcome sexual attention or advances
-* Trolling, insulting/derogatory comments, and personal or political attacks
-* Public or private harassment
-* Publishing others' private information, such as a physical or electronic address, without explicit permission
-* Other conduct which could reasonably be considered inappropriate in a professional setting
-
-## Our Responsibilities
-
-Project maintainers are responsible for clarifying the standards of acceptable behavior and are expected to take appropriate and fair corrective action in response to any instances of unacceptable behavior.
-
-Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, or to ban temporarily or permanently any contributor for other behaviors that they deem inappropriate, threatening, offensive, or harmful.
-
-## Scope
-
-This Code of Conduct applies both within project spaces and in public spaces when an individual is representing the project or its community. Examples of representing a project or community include using an official project e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event. Representation of a project may be further defined and clarified by project maintainers.
-
-## Enforcement
-
-Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by contacting the project team. All complaints will be reviewed and investigated and will result in a response that is deemed necessary and appropriate to the circumstances. The project team is obligated to maintain confidentiality with regard to the reporter of an incident. Further details of specific enforcement policies may be posted separately.
-
-Project maintainers who do not follow or enforce the Code of Conduct in good faith may face temporary or permanent repercussions as determined by other members of the project's leadership.
-
-## Attribution
-
-This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4, available at [https://contributor-covenant.org/version/1/4][version]
-
-[homepage]: https://contributor-covenant.org
-[version]: https://contributor-covenant.org/version/1/4/

.github/CONTRIBUTING.md

@@ -25,7 +25,7 @@ The following is a set of guidelines for contributing to capa and its packages,
 
 ## Code of Conduct
 
-This project and everyone participating in it is governed by the [Capa Code of Conduct](CODE_OF_CONDUCT.md). By participating, you are expected to uphold this code. Please report unacceptable behavior to the maintainers.
+This project follows [Google's Open Source Community Guidelines](https://opensource.google/conduct).
 
 ## What should I know before I get started?
 
@@ -168,15 +168,17 @@ While the prerequisites above must be satisfied prior to having your pull reques
 
 ### Contributor License Agreement
 
-Contributions to this project must be accompanied by a Contributor License
+Contributions to this project must be accompanied by a
-Agreement. You (or your employer) retain the copyright to your contribution,
+[Contributor License Agreement](https://cla.developers.google.com/about) (CLA).
-this simply gives us permission to use and redistribute your contributions as
+You (or your employer) retain the copyright to your contribution; this simply
-part of the project. Head over to <https://cla.developers.google.com/> to see
+gives us permission to use and redistribute your contributions as part of the
-your current agreements on file or to sign a new one.
+project.
 
-You generally only need to submit a CLA once, so if you've already submitted one
+If you or your current employer have already signed the Google CLA (even if it
-(even if it was for a different project), you probably don't need to do it
+was for a different project), you probably don't need to do it again.
-again.
+
+Visit <https://cla.developers.google.com/> to see your current agreements or to
+sign a new one.
 
 ## Styleguides
 

.github/ISSUE_TEMPLATE/bug_report.md

@@ -10,8 +10,8 @@ We use submodules to separate code, rules and test data. If your issue is relate
 # Have you checked that your issue isn't already filed?
 Please search if there is a similar issue at https://github.com/mandiant/capa/issues. If there is already a similar issue, please add more details there instead of opening a new one.
 
-# Have you read capa's Code of Conduct?
+# Have you read Google's Code of Conduct?
-By filing an Issue, you are expected to comply with it, including treating everyone with respect: https://github.com/mandiant/capa/blob/master/.github/CODE_OF_CONDUCT.md
+By filing an issue, you are expected to comply with it, including treating everyone with respect: https://opensource.google/conduct
 
 # Have you read capa's CONTRIBUTING guide?
 It contains helpful information about how to contribute to capa. Check https://github.com/mandiant/capa/blob/master/.github/CONTRIBUTING.md#reporting-bugs

.github/ISSUE_TEMPLATE/feature_request.md

@@ -10,8 +10,8 @@ We use submodules to separate code, rules and test data. If your issue is relate
 # Have you checked that your issue isn't already filed?
 Please search if there is a similar issue at https://github.com/mandiant/capa/issues. If there is already a similar issue, please add more details there instead of opening a new one.
 
-# Have you read capa's Code of Conduct?
+# Have you read Google's Code of Conduct?
-By filing an Issue, you are expected to comply with it, including treating everyone with respect: https://github.com/mandiant/capa/blob/master/.github/CODE_OF_CONDUCT.md
+By filing an issue, you are expected to comply with it, including treating everyone with respect: https://opensource.google/conduct
 
 # Have you read capa's CONTRIBUTING guide?
 It contains helpful information about how to contribute to capa. Check https://github.com/mandiant/capa/blob/master/.github/CONTRIBUTING.md#suggesting-enhancements

.github/dependabot.yml

@@ -4,3 +4,6 @@ updates:
     directory: "/"
     schedule:
       interval: "weekly"
+    ignore:
+      - dependency-name: "*"
+        update-types: ["version-update:semver-patch"]

.github/flake8.ini

@@ -40,4 +40,4 @@ per-file-ignores =
 
 copyright-check = True
 copyright-min-file-size = 1 
-copyright-regexp = Copyright \(C\) \d{4} Mandiant, Inc. All Rights Reserved.
+copyright-regexp = Copyright \d{4} Google LLC

.github/mypy/mypy.ini

@@ -1,8 +1,5 @@
 [mypy]
 
-[mypy-tqdm.*]
-ignore_missing_imports = True
-
 [mypy-ruamel.*]
 ignore_missing_imports = True
 

.github/pyinstaller/hooks/hook-vivisect.py

@@ -1,4 +1,16 @@
-# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
+# Copyright 2020 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
 
 from PyInstaller.utils.hooks import copy_metadata
 

.github/pyinstaller/pyinstaller.spec

@@ -1,8 +1,20 @@
 # -*- mode: python -*-
-# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
+# Copyright 2020 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 import sys
 
-import wcwidth
 import capa.rules.cache
 
 from pathlib import Path
@@ -29,13 +41,6 @@ a = Analysis(
         ("../../rules", "rules"),
         ("../../sigs", "sigs"),
         ("../../cache", "cache"),
-        # capa.render.default uses tabulate that depends on wcwidth.
-        # it seems wcwidth uses a json file `version.json`
-        # and this doesn't get picked up by pyinstaller automatically.
-        # so we manually embed the wcwidth resources here.
-        #
-        # ref: https://stackoverflow.com/a/62278462/87207
-        (Path(wcwidth.__file__).parent, "wcwidth"),
     ],
     # when invoking pyinstaller from the project root,
     # this gets run from the project root.
@@ -48,11 +53,6 @@ a = Analysis(
         "tkinter",
         "_tkinter",
         "Tkinter",
-        # tqdm provides renderers for ipython,
-        # however, this drags in a lot of dependencies.
-        # since we don't spawn a notebook, we can safely remove these.
-        "IPython",
-        "ipywidgets",
         # these are pulled in by networkx
         # but we don't need to compute the strongly connected components.
         "numpy",
@@ -70,7 +70,10 @@ a = Analysis(
         "qt5",
         "pyqtwebengine",
         "pyasn1",
+        # don't pull in Binary Ninja/IDA bindings that should
+        # only be installed locally.
         "binaryninja",
+        "ida",
     ],
 )
 

.github/ruff.toml

@@ -1,16 +1,16 @@
 # Enable the pycodestyle (`E`) and Pyflakes (`F`) rules by default.
 # Unlike Flake8, Ruff doesn't enable pycodestyle warnings (`W`) or
 # McCabe complexity (`C901`) by default.
-select = ["E", "F"]
+lint.select = ["E", "F"]
 
 # Allow autofix for all enabled rules (when `--fix`) is provided.
-fixable = ["ALL"]
+lint.fixable = ["ALL"]
-unfixable = []
+lint.unfixable = []
 
 # E402 module level import not at top of file
 # E722 do not use bare 'except'
 # E501 line too long
-ignore = ["E402", "E722", "E501"]
+lint.ignore = ["E402", "E722", "E501"]
 
 line-length = 120
 

.github/workflows/build.yml

@@ -3,6 +3,10 @@ name: build
 on:
   pull_request:
     branches: [ master ]
+    paths-ignore:
+      - 'web/**'
+      - 'doc/**'
+      - '**.md'
   release:
     types: [edited, published]
 
@@ -17,26 +21,25 @@ jobs:
       # set to false for debugging
       fail-fast: true
       matrix:
-        # using Python 3.8 to support running across multiple operating systems including Windows 7
         include:
           - os: ubuntu-20.04
             # use old linux so that the shared library versioning is more portable
             artifact_name: capa
             asset_name: linux
-            python_version: 3.8
+            python_version: '3.10'
           - os: ubuntu-20.04
             artifact_name: capa
-            asset_name: linux-py311
+            asset_name: linux-py312
-            python_version: 3.11
+            python_version: '3.12'
           - os: windows-2019
             artifact_name: capa.exe
             asset_name: windows
-            python_version: 3.8
+            python_version: '3.10'
-          - os: macos-11
+          - os: macos-13
             # use older macOS for assumed better portability
             artifact_name: capa
             asset_name: macos
-            python_version: 3.8
+            python_version: '3.10'
     steps:
       - name: Checkout capa
         uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
@@ -84,7 +87,7 @@ jobs:
             asset_name: linux
           - os: ubuntu-22.04
             artifact_name: capa
-            asset_name: linux-py311
+            asset_name: linux-py312
           - os: windows-2022
             artifact_name: capa.exe
             asset_name: windows
@@ -103,14 +106,14 @@ jobs:
     # upload zipped binaries to Release page
     if: github.event_name == 'release'
     name: zip and upload ${{ matrix.asset_name }}
-    runs-on: ubuntu-20.04
+    runs-on: ubuntu-latest
     needs: [build]
     strategy:
       matrix:
         include:
           - asset_name: linux
             artifact_name: capa
-          - asset_name: linux-py311
+          - asset_name: linux-py312
             artifact_name: capa
           - asset_name: windows
             artifact_name: capa.exe

.github/workflows/changelog.yml

@@ -13,8 +13,11 @@ permissions:
 jobs:
   check_changelog:
     # no need to check for dependency updates via dependabot
-    if: github.actor != 'dependabot[bot]' && github.actor != 'dependabot-preview[bot]'
+    # github.event.pull_request.user.login refers to PR author
-    runs-on: ubuntu-20.04
+    if: |
+      github.event.pull_request.user.login != 'dependabot[bot]' &&
+      github.event.pull_request.user.login != 'dependabot-preview[bot]'
+    runs-on: ubuntu-latest
     env:
       NO_CHANGELOG: '[x] No CHANGELOG update needed'
     steps:

.github/workflows/publish.yml

@@ -21,7 +21,7 @@ jobs:
       - name: Set up Python
         uses: actions/setup-python@0a5c61591373683505ea898e09a3ea4f39ef2b9c # v5.0.0
         with:
-          python-version: '3.8'
+          python-version: '3.10'
       - name: Install dependencies
         run: |
           python -m pip install --upgrade pip

.github/workflows/tag.yml

@@ -9,7 +9,7 @@ permissions: read-all
 jobs:
   tag:
     name: Tag capa rules
-    runs-on: ubuntu-20.04
+    runs-on: ubuntu-latest
     steps:
     - name: Checkout capa-rules
       uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1

.github/workflows/tests.yml

@@ -1,10 +1,22 @@
 name: CI
 
+# tests.yml workflow will run for all changes except:
+# any file or directory under web/ or doc/
+# any Markdown (.md) file anywhere in the repository
+
 on:
   push:
     branches: [ master ]
+    paths-ignore:
+      - 'web/**'
+      - 'doc/**'
+      - '**.md'
   pull_request:
     branches: [ master ]
+    paths-ignore:
+      - 'web/**'
+      - 'doc/**'
+      - '**.md'
 
 permissions: read-all
 
@@ -14,7 +26,7 @@ env:
 
 jobs:
   changelog_format:
-    runs-on: ubuntu-20.04
+    runs-on: ubuntu-22.04
     steps:
     - name: Checkout capa
       uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
@@ -25,15 +37,15 @@ jobs:
         if [ $number != 1 ]; then exit 1; fi
 
   code_style:
-    runs-on: ubuntu-20.04
+    runs-on: ubuntu-22.04
     steps:
     - name: Checkout capa
       uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
     # use latest available python to take advantage of best performance
-    - name: Set up Python 3.11
+    - name: Set up Python 3.12
       uses: actions/setup-python@0a5c61591373683505ea898e09a3ea4f39ef2b9c # v5.0.0
       with:
-        python-version: "3.11"
+        python-version: "3.12"
     - name: Install dependencies
       run: |
         pip install -r requirements.txt
@@ -52,16 +64,16 @@ jobs:
       run: pre-commit run deptry --hook-stage manual
 
   rule_linter:
-    runs-on: ubuntu-20.04
+    runs-on: ubuntu-22.04
     steps:
     - name: Checkout capa with submodules
       uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
       with:
         submodules: recursive
-    - name: Set up Python 3.11
+    - name: Set up Python 3.12
       uses: actions/setup-python@0a5c61591373683505ea898e09a3ea4f39ef2b9c # v5.0.0
       with:
-        python-version: "3.11"
+        python-version: "3.12"
     - name: Install capa
       run: |
         pip install -r requirements.txt
@@ -76,17 +88,17 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        os: [ubuntu-20.04, windows-2019, macos-11]
+        os: [ubuntu-20.04, windows-2019, macos-13]
         # across all operating systems
-        python-version: ["3.8", "3.11"]
+        python-version: ["3.10", "3.11"]
         include:
           # on Ubuntu run these as well
-          - os: ubuntu-20.04
-            python-version: "3.8"
-          - os: ubuntu-20.04
-            python-version: "3.9"
           - os: ubuntu-20.04
             python-version: "3.10"
+          - os: ubuntu-20.04
+            python-version: "3.11"
+          - os: ubuntu-20.04
+            python-version: "3.12"
     steps:
     - name: Checkout capa with submodules
       uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
@@ -119,7 +131,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python-version: ["3.8", "3.11"]
+        python-version: ["3.10", "3.11"]
     steps:
     - name: Checkout capa with submodules
       # do only run if BN_SERIAL is available, have to do this in every step, see https://github.com/orgs/community/discussions/26726#discussioncomment-3253118
@@ -161,7 +173,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python-version: ["3.8", "3.11"]
+        python-version: ["3.10", "3.11"]
         java-version: ["17"]
         ghidra-version: ["11.0.1"]
         public-version: ["PUBLIC_20240130"] # for ghidra releases

.github/workflows/web-deploy.yml

@@ -0,0 +1,134 @@
+name: deploy web to GitHub Pages
+
+on:
+  push:
+    branches: [ master ]
+    paths:
+      - 'web/**'
+
+  # Allows to run this workflow manually from the Actions tab
+  workflow_dispatch:
+
+# Sets the GITHUB_TOKEN permissions to allow deployment to GitHub Pages
+permissions:
+  contents: read
+  pages: write
+  id-token: write
+
+# Allow one concurrent deployment
+concurrency:
+  group: 'pages'
+  cancel-in-progress: true
+
+jobs:
+  build-landing-page:
+    name: Build landing page
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+      - uses: actions/upload-artifact@v4
+        with:
+          name: landing-page
+          path: './web/public'
+
+  build-explorer:
+    name: Build capa Explorer Web
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+        with:
+          submodules: 'recursive'
+          fetch-depth: 1
+          show-progress: true
+      - name: Set up Node
+        uses: actions/setup-node@0a44ba7841725637a19e28fa30b79a866c81b0a6 # v4.0.4
+        with:
+          node-version: 20
+          cache: 'npm'
+          cache-dependency-path: './web/explorer/package-lock.json'
+      - name: Install dependencies
+        run: npm ci
+        working-directory: ./web/explorer
+      - name: Generate release bundle
+        run: npm run build:bundle
+        working-directory: ./web/explorer
+      - name: Zip release bundle
+        run: zip -r public/capa-explorer-web.zip capa-explorer-web
+        working-directory: ./web/explorer
+      - name: Build
+        run: npm run build
+        working-directory: ./web/explorer
+      - uses: actions/upload-artifact@v4
+        with:
+          name: explorer
+          path: './web/explorer/dist'
+
+  build-rules:
+    name: Build rules site
+    runs-on: ubuntu-latest
+    steps:
+      - name: Check out the repository
+        uses: actions/checkout@v4
+        with:
+          submodules: 'recursive'
+          # full depth so that capa-rules has a full history
+          # and we can construct a timeline of rule updates.
+          fetch-depth: 0
+      - name: Set up Python
+        uses: actions/setup-python@0a5c61591373683505ea898e09a3ea4f39ef2b9c # v5.0.0
+        with:
+          python-version: '3.12'
+      - uses: extractions/setup-just@v2
+      - name: Install pagefind
+        uses: supplypike/setup-bin@v4
+        with:
+          uri: "https://github.com/CloudCannon/pagefind/releases/download/v1.1.0/pagefind-v1.1.0-x86_64-unknown-linux-musl.tar.gz"
+          name: "pagefind"
+          version: "1.1.0"
+      - name: Install dependencies
+        working-directory: ./web/rules
+        run: pip install -r requirements.txt
+      - name: Build the website
+        working-directory: ./web/rules
+        run: just build
+      - name: Index the website
+        working-directory: ./web/rules
+        run: pagefind --site "public"
+      # upload the build website to artifacts
+      # so that we can download and inspect, if desired.
+      - uses: actions/upload-artifact@v4
+        with:
+          name: rules
+          path: './web/rules/public'
+
+  deploy:
+    name: Deploy site to GitHub Pages
+    environment:
+      name: github-pages
+      url: ${{ steps.deployment.outputs.page_url }}
+    runs-on: ubuntu-latest
+    needs: [build-landing-page, build-explorer, build-rules]
+    steps:
+      - uses: actions/download-artifact@v4
+        with:
+          name: landing-page
+          path: './public/'
+      - uses: actions/download-artifact@v4
+        with:
+          name: explorer
+          path: './public/explorer'
+      - uses: actions/download-artifact@v4
+        with:
+          name: rules
+          path: './public/rules'
+      - name: Setup Pages
+        uses: actions/configure-pages@v4
+      - name: Upload artifact
+        uses: actions/upload-pages-artifact@v3
+        with:
+          path: './public'
+      - name: Deploy to GitHub Pages
+        id: deployment
+        uses: actions/deploy-pages@v4

.github/workflows/web-release.yml

@@ -0,0 +1,103 @@
+name: create web release
+on:
+  workflow_dispatch:
+    inputs:
+      version:
+        description: 'Version number for the release (x.x.x)'
+        required: true
+        type: string
+
+jobs:
+  run-tests:
+    uses: ./.github/workflows/web-tests.yml
+
+  build-and-release:
+    needs: run-tests
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v4
+
+    - name: Set release name
+      run: echo "RELEASE_NAME=capa-explorer-web-v${{ github.event.inputs.version }}-${GITHUB_SHA::7}" >> $GITHUB_ENV
+
+    - name: Check if release already exists
+      run: |
+        if ls web/explorer/releases/capa-explorer-web-v${{ github.event.inputs.version }}-* 1> /dev/null 2>&1; then
+          echo "::error:: A release with version ${{ github.event.inputs.version }} already exists"
+          exit 1
+        fi
+
+    - name: Set up Node.js
+      uses: actions/setup-node@0a44ba7841725637a19e28fa30b79a866c81b0a6 # v4.0.4
+      with:
+        node-version: 20
+        cache: 'npm'
+        cache-dependency-path: 'web/explorer/package-lock.json'
+
+    - name: Install dependencies
+      run: npm ci
+      working-directory: web/explorer
+
+    - name: Build offline bundle
+      run: npm run build:bundle
+      working-directory: web/explorer
+
+    - name: Compress bundle
+      run: zip -r ${{ env.RELEASE_NAME }}.zip capa-explorer-web
+      working-directory: web/explorer
+
+    - name: Create releases directory
+      run: mkdir -vp web/explorer/releases
+
+    - name: Move release to releases folder
+      run: mv web/explorer/${{ env.RELEASE_NAME }}.zip web/explorer/releases
+
+    - name: Compute release SHA256 hash
+      run: |
+        echo "RELEASE_SHA256=$(sha256sum web/explorer/releases/${{ env.RELEASE_NAME }}.zip | awk '{print $1}')" >> $GITHUB_ENV
+
+    - name: Update CHANGELOG.md
+      run: |
+        echo "## ${{ env.RELEASE_NAME }}" >> web/explorer/releases/CHANGELOG.md
+        echo "- Release Date: $(date -u '+%Y-%m-%d %H:%M:%S UTC')" >> web/explorer/releases/CHANGELOG.md
+        echo "- SHA256: ${{ env.RELEASE_SHA256 }}" >> web/explorer/releases/CHANGELOG.md
+        echo "" >> web/explorer/releases/CHANGELOG.md
+        cat web/explorer/releases/CHANGELOG.md
+
+    - name: Remove older releases
+      # keep only the latest 3 releases
+      run: ls -t capa-explorer-web-v*.zip | tail -n +4 | xargs -r rm --
+      working-directory: web/explorer/releases
+
+    - name: Stage release files
+      run: |
+        git config --local user.email "capa-dev@mandiant.com"
+        git config --local user.name "Capa Bot"
+        git add -f web/explorer/releases/${{ env.RELEASE_NAME }}.zip web/explorer/releases/CHANGELOG.md
+        git add -u web/explorer/releases/
+
+    - name: Create Pull Request
+      uses: peter-evans/create-pull-request@5e914681df9dc83aa4e4905692ca88beb2f9e91f # v7.0.5
+      with:
+        token: ${{ secrets.GITHUB_TOKEN }}
+        title: "explorer web: add release v${{ github.event.inputs.version }}"
+        body: |
+          This PR adds a new capa Explorer Web release v${{ github.event.inputs.version }}.
+
+          Release details:
+          - Name: ${{ env.RELEASE_NAME }}
+          - SHA256: ${{ env.RELEASE_SHA256 }}
+
+          This release is generated by the [web release](https://github.com/mandiant/capa/actions/workflows/web-release.yml) workflow.
+
+          - [x] No CHANGELOG update needed
+          - [x] No new tests needed
+          - [x] No documentation update needed
+        commit-message: ":robot: explorer web: add release ${{ env.RELEASE_NAME }}"
+        branch: release/web-v${{ github.event.inputs.version }}
+        add-paths: web/explorer/releases/${{ env.RELEASE_NAME }}.zip
+        base: master
+        labels: webui
+        delete-branch: true
+        committer: Capa Bot <capa-dev@mandiant.com>
+        author: Capa Bot <capa-dev@mandiant.com>

.github/workflows/web-tests.yml

@@ -0,0 +1,43 @@
+name: capa Explorer Web tests 
+
+on:
+  pull_request:
+    branches: [ master ]
+    paths:
+      - 'web/explorer/**'
+  workflow_call:  # this allows the workflow to be called by other workflows
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+
+    steps:
+    - name: Checkout
+      uses: actions/checkout@v4
+      with:
+        submodules: 'recursive'
+        fetch-depth: 1
+        show-progress: true
+
+    - name: Set up Node
+      uses: actions/setup-node@0a44ba7841725637a19e28fa30b79a866c81b0a6 # v4.0.4
+      with:
+        node-version: 20
+        cache: 'npm'
+        cache-dependency-path: 'web/explorer/package-lock.json'
+
+    - name: Install dependencies
+      run: npm ci
+      working-directory: web/explorer
+
+    - name: Lint
+      run: npm run lint
+      working-directory: web/explorer
+
+    - name: Format
+      run: npm run format:check
+      working-directory: web/explorer
+
+    - name: Run unit tests
+      run: npm run test
+      working-directory: web/explorer

.gitignore

@@ -126,3 +126,5 @@ Pipfile.lock
 .github/binja/binaryninja
 .github/binja/download_headless.py
 .github/binja/BinaryNinja-headless.zip
+justfile
+data/

.gitmodules

@@ -1,6 +1,6 @@
 [submodule "rules"]
 	path = rules
-	url = ../capa-rules.git
+	url = ../../mandiant/capa-rules.git
 [submodule "tests/data"]
 	path = tests/data
-	url = ../capa-testfiles.git
+	url = ../../mandiant/capa-testfiles.git

.justfile

@@ -0,0 +1,25 @@
+@isort:
+    pre-commit run isort --show-diff-on-failure --all-files
+
+@black:
+    pre-commit run black --show-diff-on-failure --all-files
+
+@ruff:
+    pre-commit run ruff --all-files
+
+@flake8:
+    pre-commit run flake8 --hook-stage manual --all-files
+
+@mypy:
+    pre-commit run mypy --hook-stage manual --all-files
+
+@deptry:
+    pre-commit run deptry --hook-stage manual --all-files
+
+@lint:
+    -just isort
+    -just black
+    -just ruff
+    -just flake8
+    -just mypy
+    -just deptry

.pre-commit-config.yaml

@@ -25,7 +25,7 @@ repos:
     hooks:
     -   id: isort
         name: isort
-        stages: [commit, push, manual]
+        stages: [pre-commit, pre-push, manual]
         language: system
         entry: isort
         args: 
@@ -38,6 +38,7 @@ repos:
         -   "capa/"
         -   "scripts/"
         -   "tests/"
+        -   "web/rules/scripts/"
         always_run: true
         pass_filenames: false
 
@@ -45,7 +46,7 @@ repos:
     hooks:
     -   id: black
         name: black
-        stages: [commit, push, manual]
+        stages: [pre-commit, pre-push, manual]
         language: system
         entry: black
         args: 
@@ -55,6 +56,7 @@ repos:
         -   "capa/"
         -   "scripts/"
         -   "tests/"
+        -   "web/rules/scripts/"
         always_run: true
         pass_filenames: false
 
@@ -62,7 +64,7 @@ repos:
     hooks:
     -   id: ruff
         name: ruff
-        stages: [commit, push, manual]
+        stages: [pre-commit, pre-push, manual]
         language: system
         entry: ruff
         args: 
@@ -72,6 +74,7 @@ repos:
         -   "capa/"
         -   "scripts/"
         -   "tests/"
+        -   "web/rules/scripts/"
         always_run: true
         pass_filenames: false
 
@@ -79,17 +82,18 @@ repos:
     hooks:
     -   id: flake8
         name: flake8
-        stages: [push, manual]
+        stages: [pre-push, manual]
         language: system
         entry: flake8
         args: 
         -   "--config"
         -   ".github/flake8.ini"
         -   "--extend-exclude"
-        -   "capa/render/proto/capa_pb2.py"
+        -   "capa/render/proto/capa_pb2.py,capa/features/extractors/binexport2/binexport2_pb2.py"
         -   "capa/"
         -   "scripts/"
         -   "tests/"
+        -   "web/rules/scripts/"
         always_run: true
         pass_filenames: false
 
@@ -97,7 +101,7 @@ repos:
     hooks:
     -   id: mypy
         name: mypy
-        stages: [push, manual]
+        stages: [pre-push, manual]
         language: system
         entry: mypy
         args: 
@@ -107,6 +111,7 @@ repos:
         -   "capa/"
         -   "scripts/"
         -   "tests/"
+        -   "web/rules/scripts/"
         always_run: true
         pass_filenames: false
 
@@ -114,7 +119,7 @@ repos:
     hooks:
     -   id: deptry
         name: deptry
-        stages: [push, manual]
+        stages: [pre-push, manual]
         language: system
         entry: deptry .
         always_run: true

CHANGELOG.md

@@ -6,43 +6,422 @@
 
 ### Breaking Changes
 
-### New Rules (0)
+### New Rules (15)
 
+- communication/socket/connect-socket moritz.raabe@mandiant.com joakim@intezer.com mrhafizfarhad@gmail.com
+- communication/socket/udp/connect-udp-socket mrhafizfarhad@gmail.com
+- nursery/enter-debug-mode-in-dotnet @v1bh475u
+- nursery/decrypt-data-using-tripledes-in-dotnet 0xRavenspar
+- nursery/encrypt-data-using-tripledes-in-dotnet 0xRavenspar
+- nursery/disable-system-features-via-registry-on-windows mehunhoff@google.com
+- data-manipulation/encryption/chaskey/encrypt-data-using-chaskey still@teamt5.org
+- data-manipulation/encryption/speck/encrypt-data-using-speck still@teamt5.org
+- load-code/dotnet/load-assembly-via-iassembly still@teamt5.org
+- malware-family/donut-loader/load-shellcode-via-donut still@teamt5.org
+- nursery/disable-device-guard-features-via-registry-on-windows mehunhoff@google.com
+- nursery/disable-firewall-features-via-registry-on-windows mehunhoff@google.com
+- nursery/disable-system-restore-features-via-registry-on-windows mehunhoff@google.com
+- nursery/disable-windows-defender-features-via-registry-on-windows mehunhoff@google.com
 -
 
 ### Bug Fixes
+- cape: make some fields optional @williballenthin #2631 #2632
+- lint: add WARN for regex features that contain unescaped dot #2635
+- lint: add ERROR for incomplete registry control set regex #2643
+
+### capa Explorer Web
+
+### capa Explorer IDA Pro plugin
+
+### Development
+
+### Raw diffs
+- [capa v9.1.0...master](https://github.com/mandiant/capa/compare/v9.1.0...master)
+- [capa-rules v9.1.0...master](https://github.com/mandiant/capa-rules/compare/v9.1.0...master)
+
+## v9.1.0
+
+This release improves a few aspects of dynamic analysis, relaxing our validation on fields across many CAPE versions, for example.
+It also includes an updated rule pack in which many dynamic rules make better use of the "span of calls" scope.
+
+
+### New Rules (3)
+
+- host-interaction/registry/change-registry-key-timestamp wballenthin@google.com
+- host-interaction/mutex/check-mutex-and-terminate-process-on-windows @_re_fox moritz.raabe@mandiant.com mehunhoff@google.com
+- anti-analysis/anti-forensic/clear-logs/clear-windows-event-logs-remotely 99.elad.levi@gmail.com
+
+### Bug Fixes
+- only parse CAPE fields required for analysis @mike-hunhoff #2607
+- main: render result document without needing associated rules @williballenthin #2610
+- vmray: only verify process OS and monitor IDs match @mike-hunhoff #2613
+- render: don't assume prior matches exist within a thread @mike-hunhoff #2612
+
+### Raw diffs
+- [capa v9.0.0...v9.1.0](https://github.com/mandiant/capa/compare/v9.0.0...v9.1.0)
+- [capa-rules v9.0.0...v9.1.0](https://github.com/mandiant/capa-rules/compare/v9.0.0...v9.1.0)
+
+## v9.0.0
+
+This release introduces a new scope for dynamic analysis, "span of calls",
+ that matches features against a across a sliding window of API calls within a thread.
+Its useful for identifying behaviors that span multiple API calls,
+ such as `OpenFile`/`ReadFile`/`CloseFile`, without having to analyze an entire thread, which may be very long.
+
+The release also contains a number of bug fixes and enhancements by new contributors: @v1bh475u and @dhruvak001. Welcome and thank you!
+
+### New Features
+
+- add warning for dynamic .NET samples #1864 @v1bh475u
+- add lint for detecting duplicate features in capa-rules #2250 @v1bh475u
+- add span-of-calls scope to match features against a across a sliding window of API calls within a thread @williballenthin #2532
+- add lint to catch rules that depend on other rules with impossible scope @williballenthin #2124
+
+### Breaking Changes
+
+- remove `is_static_limitation` method from `capa.rules.Rule`
+- add span-of-calls scope to rule format
+- capabilities functions return dataclasses instead of tuples
+
+### New Rules (3)
+
+- data-manipulation/encryption/rsa/encrypt-data-using-rsa-via-embedded-library @Ana06
+- data-manipulation/encryption/use-bigint-function @Ana06
+- internal/limitation/dynamic/internal-dotnet-file-limitation @v1bh475u
+
+
+### Bug Fixes
+
+- dynamic: only check file limitations for static file formats @mr-tz
+- vmray: load more analysis archives @mr-tz
+- vmray: skip non-printable strings @mike-hunhoff
+- vmray: loosen file checks to enable processing more file types @mike-hunhoff #2571
+- strings: add type hints and fix uncovered bugs @williballenthin #2555
+- elffile: handle symbols without a name @williballenthin #2553
+- project: remove pytest-cov that wasn't used @williballenthin @2491
+- replace binascii methods with native Python methods @v1bh475u #2582
+- rules: scopes can now have subscope blocks with the same scope @williballenthin #2584
+
+### capa Explorer Web
+
+### capa Explorer IDA Pro plugin
+
+### Development
+
+- license & copyright: Correct LICENSE file and improve copyright and license information headers in the source code files @Ana06
+- documentation: Improve CLA and Code of Conduct information in CONTRIBUTING @Ana06
+
+### Raw diffs
+- [capa v8.0.1...v9.0.0](https://github.com/mandiant/capa/compare/v8.0.1...v9.0.0)
+- [capa-rules v8.0.1...v9.0.0](https://github.com/mandiant/capa-rules/compare/v8.0.1...v9.0.0)
+
+## v8.0.1
+
+This point release fixes an issue with the IDAPython API to now handle IDA Pro 8.3, 8.4, and 9.0 correctly.
+
+### Bug Fixes
+
+- handle IDA 8.3/8.4 vs. 9.0 API change @mr-tz
+
+### Raw diffs
+- [capa v8.0.0...v8.0.1](https://github.com/mandiant/capa/compare/v8.0.0...v8.0.1)
+- [capa-rules v8.0.0...v8.0.1](https://github.com/mandiant/capa-rules/compare/v8.0.0...v8.0.1)
+
+## v8.0.0
+
+capa version 8 adds support for IDA Pro 9.0 (and idalib). The release comes with various improvements and bug fixes for the Binary Ninja backend (including to load with database files) -- thanks to @xusheng6.
+
+Additional bug fixes improve the dynamic and BinExport backends.
+
+capa version 8 now requires Python 3.10 or newer.
+
+Special thanks to @Tamir-K, @harshit-wadhwani, @jorik-utwente for their great contributions.
+
+### New Features
+
+- allow call as valid subscope for call scoped rules @mr-tz
+- support loading and analyzing a Binary Ninja database #2496 @xusheng6
+- vmray: record process command line details @mr-tz
+
+### Breaking Changes
+
+- remove support for Python 3.8 and use Python 3.10 as minimum now #1966 @mr-tz
+
+### New Rules (54)
+
+- nursery/get-shadow-password-file-entry-on-linux jonathanlepore@google.com
+- nursery/set-shadow-password-file-entry-on-linux jonathanlepore@google.com
+- collection/browser/get-chrome-cookiemonster still@teamt5.org
+- collection/browser/get-elevation-service-for-chromium-based-browsers still@teamt5.org
+- collection/get-steam-token still@teamt5.org
+- nursery/persist-via-application-shimming j.j.vannielen@utwente.nl
+- nursery/persist-via-bits-job j.j.vannielen@utwente.nl
+- nursery/persist-via-print-processors-registry-key j.j.vannielen@utwente.nl
+- linking/static/touchsocket/linked-against-touchsocket still@teamt5.org
+- runtime/dotnet/compiled-with-dotnet-aot still@teamt5.org
+- nursery/persist-via-errorhandler-script j.j.vannielen@utwente.nl
+- nursery/persist-via-get-variable-hijack j.j.vannielen@utwente.nl
+- nursery/persist-via-iphlpapi-dll-hijack j.j.vannielen@utwente.nl
+- nursery/persist-via-lnk-shortcut j.j.vannielen@utwente.nl
+- nursery/persist-via-powershell-profile j.j.vannielen@utwente.nl
+- nursery/persist-via-windows-accessibility-tools j.j.vannielen@utwente.nl
+- nursery/persist-via-windows-terminal-profile j.j.vannielen@utwente.nl
+- nursery/write-to-browser-extension-directory j.j.vannielen@utwente.nl
+- nursery/persist-via-aedebug-registry-key j.j.vannielen@utwente.nl
+- nursery/persist-via-amsi-registry-key j.j.vannielen@utwente.nl
+- nursery/persist-via-app-paths-registry-key j.j.vannielen@utwente.nl
+- nursery/persist-via-appcertdlls-registry-key j.j.vannielen@utwente.nl
+- nursery/persist-via-appx-registry-key j.j.vannielen@utwente.nl
+- nursery/persist-via-autodialdll-registry-key j.j.vannielen@utwente.nl
+- nursery/persist-via-autoplayhandlers-registry-key j.j.vannielen@utwente.nl
+- nursery/persist-via-bootverificationprogram-registry-key j.j.vannielen@utwente.nl
+- nursery/persist-via-code-signing-registry-key j.j.vannielen@utwente.nl
+- nursery/persist-via-com-hijack j.j.vannielen@utwente.nl
+- nursery/persist-via-command-processor-registry-key j.j.vannielen@utwente.nl
+- nursery/persist-via-contextmenuhandlers-registry-key j.j.vannielen@utwente.nl
+- nursery/persist-via-cor_profiler_path-registry-value j.j.vannielen@utwente.nl
+- nursery/persist-via-default-file-association-registry-key j.j.vannielen@utwente.nl
+- nursery/persist-via-disk-cleanup-handler-registry-key j.j.vannielen@utwente.nl
+- nursery/persist-via-dotnet-dbgmanageddebugger-registry-key j.j.vannielen@utwente.nl
+- nursery/persist-via-dotnet_startup_hooks-registry-key j.j.vannielen@utwente.nl
+- nursery/persist-via-explorer-tools-registry-key j.j.vannielen@utwente.nl
+- nursery/persist-via-filter-handlers-registry-key j.j.vannielen@utwente.nl
+- nursery/persist-via-group-policy-registry-key j.j.vannielen@utwente.nl
+- nursery/persist-via-hhctrl-com-hijack j.j.vannielen@utwente.nl
+- nursery/persist-via-htmlhelp-author-registry-key j.j.vannielen@utwente.nl
+- nursery/persist-via-image-file-execution-options-registry-key j.j.vannielen@utwente.nl
+- nursery/persist-via-lsa-registry-key j.j.vannielen@utwente.nl
+- nursery/persist-via-natural-language-registry-key j.j.vannielen@utwente.nl
+- nursery/persist-via-netsh-registry-key j.j.vannielen@utwente.nl
+- nursery/persist-via-network-provider-registry-key j.j.vannielen@utwente.nl
+- nursery/persist-via-path-registry-key j.j.vannielen@utwente.nl
+- nursery/persist-via-print-monitors-registry-key j.j.vannielen@utwente.nl
+- nursery/persist-via-rdp-startup-programs-registry-key j.j.vannielen@utwente.nl
+- nursery/persist-via-silentprocessexit-registry-key j.j.vannielen@utwente.nl
+- nursery/persist-via-telemetrycontroller-registry-key j.j.vannielen@utwente.nl
+- nursery/persist-via-timeproviders-registry-key j.j.vannielen@utwente.nl
+- nursery/persist-via-ts-initialprogram-registry-key j.j.vannielen@utwente.nl
+- nursery/persist-via-userinitmprlogonscript-registry-value j.j.vannielen@utwente.nl
+- nursery/persist-via-windows-error-reporting-registry-key j.j.vannielen@utwente.nl
+
+### Bug Fixes
+
+- extractor: fix exception when PE extractor encounters unknown architecture #2440 @Tamir-K
+- IDA Pro: rename ida to idapro module for plugin and idalib in IDA 9.0 #2453 @mr-tz
+- ghidra: fix saving of base address @mr-tz
+- binja: support loading raw x86/x86_64 shellcode #2489 @xusheng6
+- binja: fix crash when the IL of certain functions are not available. #2249 @xusheng6
+- binja: major performance improvement on the binja extractor. #1414 @xusheng6
+- cape: make Process model flexible and procmemory optional to load newest reports #2466 @mr-tz
+- binja: fix unit test failure by fixing up the analysis for file al-khaser_x64.exe_ #2507 @xusheng6
+- binja: move the stack string detection to function level #2516 @xusheng6
+- BinExport2: fix handling of incorrect thunk functions #2524 @williballenthin
+- BinExport2: more precise pruning of expressions @williballenthin
+- BinExport2: better handle weird expression trees from Ghidra #2528 #2530 @williballenthin
+
+### capa Explorer Web
+
+### capa Explorer IDA Pro plugin
+
+- fix bug preventing saving of capa results via Save button @mr-tz
+- fix saving of base address @mr-tz
+
+### Development
+- CI: use macos-13 since macos-12 is deprecated and will be removed on December 3rd, 2024 #2173 @mr-tz
+- CI: update Binary Ninja version to 4.2 #2499 @xusheng6
+
+### Raw diffs
+- [capa v7.4.0...v8.0.0](https://github.com/mandiant/capa/compare/v7.4.0...v8.0.0)
+- [capa-rules v7.4.0...v8.0.0](https://github.com/mandiant/capa-rules/compare/v7.4.0...v8.0.0)
+
+## v7.4.0
+
+The v7.4.0 capa release fixes a bug when processing VMRay analysis archives and enhances API extraction for all dynamic backends. For better terminal rendering capa now solely relies on the rich library.
+
+The standalone capa executable can now automatically detect installations of relevant third party applications and use their backends (notably, idalib and Binary Ninja). For the extra standalone Linux build we've upgraded from Python 3.11 to 3.12.
+
+Twelve new rules have been added. Thanks to all the contributors!
+
+*This is the last capa version supporting Python 3.8 and 3.9. If you have major concerns about this please reach out to us.*
+
+### New Features
+
+- add IDA v9.0 backend via idalib #2376 @williballenthin
+- locate Binary Ninja API using XDG Desktop Entries #2376 @williballenthin
+
+### New Rules (15)
+
+- nursery/access-unmanaged-com-objects-in-dotnet mehunhoff@google.com
+- nursery/implement-ui-automation-client-in-dotnet mehunhoff@google.com
+- nursery/interact-with-shortcut-via-iwshshortcut-in-dotnet mehunhoff@google.com
+- nursery/interact-with-windows-scripting-host-in-dotnet mehunhoff@google.com
+- nursery/use-dotnet-library-simplejson mehunhoff@google.com
+- nursery/use-dotnet-library-websocket-sharp mehunhoff@google.com
+- linking/runtime-linking/populate-syswhispers2-syscall-list still@teamt5.org
+- host-interaction/os/hide-shutdown-actions-via-policy still@teamt5.org
+- host-interaction/process/get-process-filename matthew.williams@mandiant.com
+- host-interaction/driver/complete-processing-asynchronous-io-request moritz.raabe@mandiant.com
+- anti-analysis/packer/nmm-protect/packed-with-nmm-protect william.ballenthin@mandiant.com
+- host-interaction/firewall/modify/access-firewall-policy-via-inetfwpolicy2 jakub.jozwiak@mandiant.com
+- host-interaction/firewall/modify/access-firewall-rule-properties-via-inetfwrule jakub.jozwiak@mandiant.com
+- host-interaction/registry/open-recentdocs-registry-key matthew.williams@mandiant.com
+
+### Bug Fixes
+
+- use Python 3.12 to build extra standalone build on Linux #2383 @williballenthin
+- bump minimum Python version to 3.8.1 to satisfy uv #2387 @williballenthin
+- vmray: collect more process information from flog.xml #2394 @mr-tz @mike-hunhoff
+- replace tabulate, tqdm, and termcolor with rich #2374 @s-ff
+- dynamic: emit complete features for A/W APIs #2409 @mike-hunhoff
+- vmray: fix backslash handling in string call arguments #2428 @mr-tz
+
+### capa Explorer Web
+- improve navigation in capa Explorer Web @s-ff #2425
+
+### Raw diffs
+
+- [capa v7.3.0...v7.4.0](https://github.com/mandiant/capa/compare/v7.3.0...v7.4.0)
+- [capa-rules v7.3.0...v7.4.0](https://github.com/mandiant/capa-rules/compare/v7.3.0...v7.4.0)
+
+## v7.3.0
+
+This release comes with the following three major enhancements:
+
+### 1. Support for VMRay sandbox analysis archives
+
+Unlock powerful malware analysis with capa's new [VMRay sandbox](https://www.vmray.com/) integration! Simply provide a VMRay analysis archive, and capa will automatically extract and match capabilities to streamline your workflow. This is the second support for the analysis of dynamic analysis results after [CAPE](https://www.mandiant.com/resources/blog/dynamic-capa-executable-behavior-cape-sandbox).
+
+### 2. Support for BinExport files generated by Ghidra
+
+[BinExport](https://github.com/google/binexport) files store disassembled data into a Protocol Buffer format. capa now supports the analysis of BinExport files generated by Ghidra. Using Ghidra and the BinExport file format users can now analyze ARM (AARCH64) ELF files targeting Android.
+
+### 3. Introducing the capa rules website
+
+You can now browse capa's default rule set at https://mandiant.github.io/capa/rules. In modern terminals the CLI capa tool hyperlinks to resources on the web, including entries on the capa rules website.
+Furthermore, https://mandiant.github.io/capa provides a landing page for the capa tool project.
+
+### Additional updates
+
+- [capa Explorer Web](https://mandiant.github.io/capa/explorer/) received several enhancements and bug fixes.
+- Support for the IDA Pro 9.0 IDAPython API while keeping compatibility to older IDA Pro versions
+- Six rules have been added and two rules have been updated
+
+Thanks to @r-sm2024 for their contribution in https://github.com/mandiant/capa/pull/2155 and their further work. And of course a big thanks to the community for reporting issues, participating in discussions, and supporting the capa tool and capa rules.
+
+### New Features
+
+- regenerate ruleset cache automatically on source change (only in dev mode) #2133 @s-ff
+- add landing page https://mandiant.github.io/capa/ @williballenthin #2310
+- add rules website https://mandiant.github.io/capa/rules @DeeyaSingh #2310
+- add .justfile @williballenthin #2325
+- dynamic: add support for VMRay dynamic sandbox traces #2208 @mike-hunhoff @r-sm2024 @mr-tz
+- cli: use modern terminal features to hyperlink to the rules website #2337 @williballenthin
+- support analyzing BinExport2 files generated by Ghidra #1950 @williballenthin @mehunhoff @mr-tz
+- add support for Android OS #1950 @williballenthin @mehunhoff @mr-tz
+- add support for aarch64 architecture via BinExport2 backend #1950 @williballenthin @mehunhoff @mr-tz
+
+### New Rules (6)
+
+- linking/static/minhook/linked-against-minhook jakub.jozwiak@mandiant.com
+- linking/static/sqlite3/linked-against-sqlcipher wballenthin@google.com
+- host-interaction/network/traffic/filter/delete-network-filter-via-wfp-api jakub.jozwiak@mandiant.com
+- host-interaction/network/traffic/filter/enumerate-network-filters-via-wfp-api jakub.jozwiak@mandiant.com
+- nursery/check-thread-suspend-count-exceeded ervinocampo@google.com
+- nursery/create-thread-bypassing-process-freeze ervinocampo@google.com
+
+### Bug Fixes
+
+- fix duplicate features shown in vverbose mode @williballenthin #2323
+- fix code path reference in linter @williballenthin #2350
+
+### capa explorer IDA Pro plugin
+
+- update IDAPython to IDA Pro 9.0 @mr-tz
+- fix byte search IDA Pro 7.5 compatibility @mr-tz #2371
+
+### Raw diffs
+
+- [capa v7.2.0...v7.3.0](https://github.com/mandiant/capa/compare/v7.2.0...v7.3.0)
+- [capa-rules v7.2.0...v7.3.0](https://github.com/mandiant/capa-rules/compare/v7.2.0...v7.3.0)
+
+## v7.2.0
+
+capa v7.2.0 introduces a first version of capa explorer web: a web-based user interface to inspect capa results using your browser. Users can inspect capa result JSON documents in an online web instance or a standalone HTML page for offline usage. capa explorer supports interactive exploring of capa results to make it easier to understand them. Users can filter, sort, and see the details of all identified capabilities. capa explorer web was worked on by @s-ff as part of a [GSoC project](https://summerofcode.withgoogle.com/programs/2024/projects/cR3hjbsq), and it is available at https://mandiant.github.io/capa/explorer/#/.
+
+This release also adds a feature extractor for output from the DRAKVUF sandbox. Now, analysts can pass the resulting `drakmon.log` file to capa and extract capabilities from the artifacts captured by the sandbox. This feature extractor will also be added to the DRAKVUF sandbox as a post-processing script, and it was worked on by @yelhamer as part of a [GSoC project](https://summerofcode.withgoogle.com/programs/2024/projects/fCnBGuEC).
+
+Additionally, we fixed several bugs handling ELF files, and added the ability to filter capa analysis by functions or processes. We also added support to the IDA Pro extractor to leverage analyst recovered API names.
+
+Special thanks to our repeat and new contributors:
+
+- @lakshayletsgo for their first contribution in https://github.com/mandiant/capa/pull/2248
+- @msm-cert for their first contribution in https://github.com/mandiant/capa/pull/2143
+- @VascoSch92 for their first contribution in https://github.com/mandiant/capa/pull/2143
+
+### New Features
+
+- webui: explore capa analysis results in a web-based UI online and offline #2224 @s-ff
+- support analyzing DRAKVUF traces #2143 @yelhamer
+- IDA extractor: extract names from dynamically resolved APIs stored in renamed global variables #2201 @Ana06
+- cli: add the ability to select which specific functions or processes to analyze @yelhamer
+
+### Breaking Changes
+
+### New Rules (5)
+
+- nursery/upload-file-to-onedrive jaredswilson@google.com ervinocampo@google.com
+- data-manipulation/encoding/base64/decode-data-using-base64-via-vbmi-lookup-table still@teamt5.org
+- communication/socket/attach-bpf-to-socket-on-linux jakub.jozwiak@mandiant.com
+- anti-analysis/anti-av/overwrite-dll-text-section-to-remove-hooks jakub.jozwiak@mandiant.com
+- nursery/delete-file-on-linux mehunhoff@google.com
+
+### Bug Fixes
+
+- elf: extract import / export symbols from stripped binaries #2096 @ygasparis
+- elf: fix handling of symbols in corrupt ELF files #2226 @williballenthin
 
 ### capa explorer IDA Pro plugin
 
 ### Development
 
+- CI: use macos-12 since macos-11 is deprecated and will be removed on June 28th, 2024 #2173 @mr-tz
+- CI: update Binary Ninja version to 4.1 and use Python 3.9 to test it #2211 @xusheng6
+- CI: update tests.yml workflow to exclude web and documentation files #2263 @s-ff
+- CI: update build.yml workflow to exclude web and documentation files #2270 @s-ff
+- CI: add web releases workflow #2455 @s-ff
+- CI: skip changelog.yml for dependabot PRs #2471
+
 ### Raw diffs
-- [capa v7.1.0...master](https://github.com/mandiant/capa/compare/v7.1.0...master)
+
-- [capa-rules v7.1.0...master](https://github.com/mandiant/capa-rules/compare/v7.1.0...master)
+- [capa v7.1.0...7.2.0](https://github.com/mandiant/capa/compare/v7.1.0...7.2.0)
+- [capa-rules v7.1.0...7.2.0](https://github.com/mandiant/capa-rules/compare/v7.1.0...7.2.0)
 
 ## v7.1.0
+
 The v7.1.0 release brings large performance improvements to capa's rule matching engine.
 Additionally, we've fixed various bugs and added new features for people using and developing capa.
 
 Special thanks to our repeat and new contributors:
-* @sjha2048 made their first contribution in https://github.com/mandiant/capa/pull/2000
+
-* @Rohit1123 made their first contribution in https://github.com/mandiant/capa/pull/1990
+- @sjha2048 made their first contribution in https://github.com/mandiant/capa/pull/2000
-* @psahithireddy made their first contribution in https://github.com/mandiant/capa/pull/2020
+- @Rohit1123 made their first contribution in https://github.com/mandiant/capa/pull/1990
-* @Atlas-64 made their first contribution in https://github.com/mandiant/capa/pull/2018
+- @psahithireddy made their first contribution in https://github.com/mandiant/capa/pull/2020
-* @s-ff made their first contribution in https://github.com/mandiant/capa/pull/2011
+- @Atlas-64 made their first contribution in https://github.com/mandiant/capa/pull/2018
-* @samadpls made their first contribution in https://github.com/mandiant/capa/pull/2024
+- @s-ff made their first contribution in https://github.com/mandiant/capa/pull/2011
-* @acelynnzhang made their first contribution in https://github.com/mandiant/capa/pull/2044
+- @samadpls made their first contribution in https://github.com/mandiant/capa/pull/2024
-* @RainRat made their first contribution in https://github.com/mandiant/capa/pull/2058
+- @acelynnzhang made their first contribution in https://github.com/mandiant/capa/pull/2044
-* @ReversingWithMe made their first contribution in https://github.com/mandiant/capa/pull/2093
+- @RainRat made their first contribution in https://github.com/mandiant/capa/pull/2058
-* @malwarefrank made their first contribution in https://github.com/mandiant/capa/pull/2037
+- @ReversingWithMe made their first contribution in https://github.com/mandiant/capa/pull/2093
+- @malwarefrank made their first contribution in https://github.com/mandiant/capa/pull/2037
 
 ### New Features
+
 - Emit "dotnet" as format to ResultDocument when processing .NET files #2024 @samadpls
 - ELF: detect OS from statically-linked Go binaries #1978 @williballenthin
 - add function in capa/helpers to load plain and compressed JSON reports #1883 @Rohit1123
 - document Antivirus warnings and VirusTotal false positive detections #2028 @RionEV @mr-tz
 - Add json to sarif conversion script @reversingwithme
-- render maec/* fields #843 @s-ff
+- render maec/\* fields #843 @s-ff
 - replace Halo spinner with Rich #2086 @s-ff
 - optimize rule matching #2080 @williballenthin
 - add aarch64 as a valid architecture #2144 mehunhoff@google.com @williballenthin
@@ -85,6 +464,7 @@ Special thanks to our repeat and new contributors:
 - cape: support more report formats #2035 @mr-tz
 
 ### capa explorer IDA Pro plugin
+
 - replace deprecated IDA API find_binary with bin_search #1606 @s-ff
 
 ### Development
@@ -98,6 +478,7 @@ Special thanks to our repeat and new contributors:
 - add deptry support #1497 @s-ff
 
 ### Raw diffs
+
 - [capa v7.0.1...v7.1.0](https://github.com/mandiant/capa/compare/v7.0.1...v7.1.0)
 - [capa-rules v7.0.1...v7.1.0](https://github.com/mandiant/capa-rules/compare/v7.0.1...v7.1.0)
 
@@ -110,10 +491,12 @@ This release fixes a circular import error when using capa as a library.
 - fix potentially circular import errors #1969 @williballenthin
 
 ### Raw diffs
+
 - [capa v7.0.0...v7.0.1](https://github.com/mandiant/capa/compare/v7.0.0...v7.0.1)
 - [capa-rules v7.0.0...v7.0.1](https://github.com/mandiant/capa-rules/compare/v7.0.0...v7.0.1)
 
 ## v7.0.0
+
 This is the v7.0.0 release of capa which was mainly worked on during the Google Summer of Code (GSoC) 2023. A huge
 shoutout to our GSoC contributors @colton-gabertan and @yelhamer for their amazing work.
 
@@ -189,6 +572,7 @@ Also, a big thanks to the other contributors: @aaronatp, @Aayush-Goel-04, @bkoju
 - host-interaction/process/inject/process-ghostly-hollowing sara.rincon@mandiant.com
 
 ### Bug Fixes
+
 - ghidra: fix `ints_to_bytes` performance #1761 @mike-hunhoff
 - binja: improve function call site detection @xusheng6
 - binja: use `binaryninja.load` to open files @xusheng6
@@ -199,12 +583,15 @@ Also, a big thanks to the other contributors: @aaronatp, @Aayush-Goel-04, @bkoju
 - remove unnecessary scripts/vivisect-py2-vs-py3.sh file #1949 @JCoonradt
 
 ### capa explorer IDA Pro plugin
+
 - various integration updates and minor bug fixes
 
 ### Development
+
 - update ATT&CK/MBC data for linting #1932 @mr-tz
 
 #### Developer Notes
+
 With this new release, many classes and concepts have been split up into static (mostly identical to the
 prior implementations) and dynamic ones. For example, the legacy FeatureExtractor class has been renamed to
 StaticFeatureExtractor and the DynamicFeatureExtractor has been added.
@@ -225,6 +612,7 @@ format and backend is more consistent. We've documented that the input file is n
 (cape/freeze/etc.) inputs are not actually the sample.
 
 ### Raw diffs
+
 - [capa v6.1.0...v7.0.0](https://github.com/mandiant/capa/compare/v6.1.0...v7.0.0)
 - [capa-rules v6.1.0...v7.0.0](https://github.com/mandiant/capa-rules/compare/v6.1.0...v7.0.0)
 
@@ -238,6 +626,7 @@ You could use this script to find opportunities for new rules.
 Speaking of new rules, we have eight additions, coming from Ronnie, Jakub, Moritz, Ervin, and still@teamt5.org!
 
 ### New Features
+
 - ELF: implement import and export name extractor #1607 #1608 @Aayush-Goel-04
 - bump pydantic from 1.10.9 to 2.1.1 #1582 @Aayush-Goel-04
 - develop script to highlight features not used during matching #331 @Aayush-Goel-04
@@ -263,9 +652,11 @@ Speaking of new rules, we have eight additions, coming from Ronnie, Jakub, Morit
 - pytest: don't mark IDA tests as pytest tests #1719 @williballenthin
 
 ### capa explorer IDA Pro plugin
+
 - fix unhandled exception when resolving rule path #1693 @mike-hunhoff
 
 ### Raw diffs
+
 - [capa v6.0.0...v6.1.0](https://github.com/mandiant/capa/compare/v6.0.0...v6.1.0)
 - [capa-rules v6.0.0...v6.1.0](https://github.com/mandiant/capa-rules/compare/v6.0.0...v6.1.0)
 
@@ -276,12 +667,14 @@ capa v6.0 brings many bug fixes and quality improvements, including 64 rule upda
 For those that use capa as a library, we've introduced some limited breaking changes that better represent data types (versus less-structured data like dictionaries and strings). With the recent deprecation, we've also dropped support for Python 3.7.
 
 ### New Features
+
 - add script to detect feature overlap between new and existing capa rules [#1451](https://github.com/mandiant/capa/issues/1451) [@Aayush-Goel-04](https://github.com/aayush-goel-04)
 - extract forwarded exports from PE files #1624 @williballenthin
 - extract function and API names from ELF symtab entries @yelhamer https://github.com/mandiant/capa-rules/issues/736
 - use fancy box drawing characters for default output #1586 @williballenthin
 
 ### Breaking Changes
+
 - use a class to represent Metadata (not dict) #1411 @Aayush-Goel-04 @manasghandat
 - use pathlib.Path to represent file paths #1534 @Aayush-Goel-04
 - Python 3.8 is now the minimum supported Python version #1578 @williballenthin
@@ -316,6 +709,7 @@ For those that use capa as a library, we've introduced some limited breaking cha
 - linking/runtime-linking/resolve-function-by-brute-ratel-badger-hash jakub.jozwiak@mandiant.com
 
 ### Bug Fixes
+
 - extractor: add a Binary Ninja test that asserts its version #1487 @xusheng6
 - extractor: update Binary Ninja stack string detection after the new constant outlining feature #1473 @xusheng6
 - extractor: update vivisect Arch extraction #1334 @mr-tz
@@ -336,6 +730,7 @@ For those that use capa as a library, we've introduced some limited breaking cha
 ### capa explorer IDA Pro plugin
 
 ### Development
+
 - update ATT&CK/MBC data for linting #1568 @mr-tz
 - log time taken to analyze each function #1290 @williballenthin
 - tests: make fixture available via conftest.py #1592 @williballenthin
@@ -343,12 +738,13 @@ For those that use capa as a library, we've introduced some limited breaking cha
 - migrate to pyproject.toml #1301 @williballenthin
 - use [pre-commit](https://pre-commit.com/) to invoke linters #1579 @williballenthin
 
-
 ### Raw diffs
+
 - [capa v5.1.0...v6.0.0](https://github.com/mandiant/capa/compare/v5.1.0...v6.0.0)
 - [capa-rules v5.1.0...v6.0.0](https://github.com/mandiant/capa-rules/compare/v5.1.0...v6.0.0)
 
 ## v5.1.0
+
 capa version 5.1.0 adds a Protocol Buffers (protobuf) format for result documents. Additionally, the [Vector35](https://vector35.com/) team contributed a new feature extractor using Binary Ninja. Other new features are a new CLI flag to override the detected operating system, functionality to read and render existing result documents, and an output color format that's easier to read.
 
 Over 25 capa rules have been added and improved.
@@ -356,6 +752,7 @@ Over 25 capa rules have been added and improved.
 Thanks for all the support, especially to @xusheng6, @captainGeech42, @ggold7046, @manasghandat, @ooprathamm, @linpeiyu164, @yelhamer, @HongThatCong, @naikordian, @stevemk14ebr, @emtuls, @raymondlleong, @bkojusner, @joren485, and everyone else who submitted bugs and provided feedback!
 
 ### New Features
+
 - add protobuf format for result documents #1219 @williballenthin @mr-tz
 - extractor: add Binary Ninja feature extractor @xusheng6
 - new cli flag `--os` to override auto-detected operating system for a sample @captainGeech42
@@ -393,6 +790,7 @@ Thanks for all the support, especially to @xusheng6, @captainGeech42, @ggold7046
 - nursery/contain-a-thread-local-storage-tls-section-in-dotnet michael.hunhoff@mandiant.com
 
 ### Bug Fixes
+
 - extractor: interface of cache modified to prevent extracting file and global features multiple times @stevemk14ebr
 - extractor: removed '.dynsym' as the library name for ELF imports #1318 @stevemk14ebr
 - extractor: fix vivisect loop detection corner case #1310 @mr-tz
@@ -400,16 +798,18 @@ Thanks for all the support, especially to @xusheng6, @captainGeech42, @ggold7046
 - extractor: fix IDA and vivisect string and bytes features overlap and tests #1327 #1336 @xusheng6
 
 ### capa explorer IDA Pro plugin
+
 - rule generator plugin now loads faster when jumping between functions @stevemk14ebr
 - fix exception when plugin loaded in IDA hosted under idat #1341 @mike-hunhoff
 - improve embedded PE detection performance and reduce FP potential #1344 @mike-hunhoff
 
 ### Raw diffs
+
 - [capa v5.0.0...v5.1.0](https://github.com/mandiant/capa/compare/v5.0.0...v5.1.0)
 - [capa-rules v5.0.0...v5.1.0](https://github.com/mandiant/capa-rules/compare/v5.0.0...v5.1.0)
 
-
 ## v5.0.0 (2023-02-08)
+
 This capa version comes with major improvements and additions to better handle .NET binaries. To showcase this we've updated and added over 30 .NET rules.
 
 Additionally, capa now caches its rule set for better performance. The capa explorer also caches its analysis results, so that multiple IDA Pro or plugin invocations don't need to repeat the same analysis.
@@ -421,6 +821,7 @@ Other improvements to highlight include better ELF OS detection, various renderi
 Thanks for all the support, especially to @jsoref, @bkojusner, @edeca, @richardweiss80, @joren485, @ryantxu1, @mwilliams31, @anushkavirgaonkar, @MalwareMechanic, @Still34, @dzbeck, @johnk3r, and everyone else who submitted bugs and provided feedback!
 
 ### New Features
+
 - verify rule metadata format on load #1160 @mr-tz
 - dotnet: emit property features #1168 @anushkavirgaonkar
 - dotnet: emit API features for objects created via the newobj instruction #1186 @mike-hunhoff
@@ -435,6 +836,7 @@ Thanks for all the support, especially to @jsoref, @bkojusner, @edeca, @richardw
 - update ATT&CK/MBC data for linting #1297 @mr-tz
 
 ### Breaking Changes
+
 - remove SMDA backend #1062 @williballenthin
 - error return codes are now positive numbers #1269 @mr-tz
 
@@ -514,6 +916,7 @@ Thanks for all the support, especially to @jsoref, @bkojusner, @edeca, @richardw
 - nursery/unmanaged-call-via-dynamic-pinvoke-in-dotnet michael.hunhoff@mandiant.com
 
 ### Bug Fixes
+
 - render: convert feature attributes to aliased dictionary for vverbose #1152 @mike-hunhoff
 - decouple Token dependency / extractor and features #1139 @mr-tz
 - update pydantic model to guarantee type coercion #1176 @mike-hunhoff
@@ -532,6 +935,7 @@ Thanks for all the support, especially to @jsoref, @bkojusner, @edeca, @richardw
 - extractor: don't extract byte features for strings #1293 @mr-tz
 
 ### capa explorer IDA Pro plugin
+
 - fix: display instruction items #1154 @mr-tz
 - fix: accept only plaintext pasted content #1194 @williballenthin
 - fix: UnboundLocalError #1217 @williballenthin
@@ -546,27 +950,30 @@ Thanks for all the support, especially to @jsoref, @bkojusner, @edeca, @richardw
 - cache capa results across IDA sessions #1279 @mr-tz
 
 ### Raw diffs
+
 - [capa v4.0.1...v5.0.0](https://github.com/mandiant/capa/compare/v4.0.1...v5.0.0)
 - [capa-rules v4.0.1...v5.0.0](https://github.com/mandiant/capa-rules/compare/v4.0.1...v5.0.0)
 
-
 ## v4.0.1 (2022-08-15)
+
 Some rules contained invalid metadata fields that caused an error when rendering rule hits. We've updated all rules and enhanced the rule linter to catch such issues.
 
 ### New Rules (1)
 
 - anti-analysis/obfuscation/obfuscated-with-vs-obfuscation jakub.jozwiak@mandiant.com
 
-
 ### Bug Fixes
+
 - linter: use pydantic to validate rule metadata #1141 @mike-hunhoff
 - build binaries using PyInstaller no longer overwrites functions in version.py #1136 @mr-tz
 
 ### Raw diffs
+
 - [capa v4.0.0...v4.0.1](https://github.com/mandiant/capa/compare/v4.0.0...v4.0.1)
 - [capa-rules v4.0.0...v4.0.1](https://github.com/mandiant/capa-rules/compare/v4.0.0...v4.0.1)
 
 ## v4.0.0 (2022-08-10)
+
 Version 4 adds support for analyzing .NET executables. capa will autodetect .NET modules, or you can explicitly invoke the new feature extractor via `--format dotnet`. We've also extended the rule syntax for .NET features including `namespace` and `class`.
 
 Additionally, new `instruction` scope and `operand` features enable users to create more explicit rules. These features are not backwards compatible. We removed the previously used `/x32` and `/x64` flavors of number and operand features.
@@ -575,9 +982,9 @@ We updated 49 existing rules and added 22 new rules leveraging these new feature
 
 More breaking changes include updates to the JSON results document, freeze file format schema (now format version v2), and the internal handling of addresses.
 
-Thanks for all the support, especially to @htnhan, @jtothej, @sara-rn, @anushkavirgaonkar, and @_re_fox!
+Thanks for all the support, especially to @htnhan, @jtothej, @sara-rn, @anushkavirgaonkar, and @\_re_fox!
 
-*Deprecation warning: v4.0 will be the last capa version to support the SMDA backend.*
+_Deprecation warning: v4.0 will be the last capa version to support the SMDA backend._
 
 ### New Features
 
@@ -618,11 +1025,11 @@ Deprecation notice: as described in [#937](https://github.com/mandiant/capa/issu
 - anti-analysis/packer/huan/packed-with-huan jakub.jozwiak@mandiant.com
 - nursery/execute-dotnet-assembly anushka.virgaonkar@mandiant.com
 - nursery/invoke-dotnet-assembly-method anushka.virgaonkar@mandiant.com
-- collection/screenshot/capture-screenshot-via-keybd-event @_re_fox
+- collection/screenshot/capture-screenshot-via-keybd-event @\_re_fox
-- collection/browser/gather-chrome-based-browser-login-information @_re_fox
+- collection/browser/gather-chrome-based-browser-login-information @\_re_fox
 - nursery/power-down-monitor michael.hunhoff@mandiant.com
-- nursery/hash-data-using-aphash @_re_fox
+- nursery/hash-data-using-aphash @\_re_fox
-- nursery/hash-data-using-jshash @_re_fox
+- nursery/hash-data-using-jshash @\_re_fox
 - host-interaction/file-system/files/list/enumerate-files-on-windows moritz.raabe@mandiant.com anushka.virgaonkar@mandiant.com
 - nursery/check-clipboard-data anushka.virgaonkar@mandiant.com
 - nursery/clear-clipboard-data anushka.virgaonkar@mandiant.com
@@ -634,16 +1041,18 @@ Deprecation notice: as described in [#937](https://github.com/mandiant/capa/issu
 - nursery/send-keystrokes anushka.virgaonkar@mandiant.com
 - nursery/send-request-in-dotnet anushka.virgaonakr@mandiant.com
 - nursery/terminate-process-by-name-in-dotnet anushka.virgaonkar@mandiant.com
-- nursery/hash-data-using-rshash @_re_fox
+- nursery/hash-data-using-rshash @\_re_fox
 - persistence/authentication-process/act-as-credential-manager-dll jakub.jozwiak@mandiant.com
 - persistence/authentication-process/act-as-password-filter-dll jakub.jozwiak@mandiant.com
 
 ### Bug Fixes
-- improve handling _ prefix compile/link artifact #924 @mike-hunhoff
+
+- improve handling \_ prefix compile/link artifact #924 @mike-hunhoff
 - better detect OS in ELF samples #988 @williballenthin
 - display number feature zero in vverbose #1097 @mike-hunhoff
 
 ### capa explorer IDA Pro plugin
+
 - improve file format extraction #918 @mike-hunhoff
 - remove decorators added by IDA to ELF imports #919 @mike-hunhoff
 - bug fixes for Address abstraction #1091 @mike-hunhoff
@@ -651,10 +1060,12 @@ Deprecation notice: as described in [#937](https://github.com/mandiant/capa/issu
 ### Development
 
 ### Raw diffs
+
 - [capa v3.2.0...v4.0.0](https://github.com/mandiant/capa/compare/v3.2.0...master)
 - [capa-rules v3.2.0...v4.0.0](https://github.com/mandiant/capa-rules/compare/v3.2.0...master)
 
 ## v3.2.1 (2022-06-06)
+
 This out-of-band release bumps the SMDA dependency version to enable installation on Python 3.10.
 
 ### Bug Fixes
@@ -662,10 +1073,12 @@ This out-of-band release bumps the SMDA dependency version to enable installatio
 - update SMDA dependency @mike-hunhoff #922
 
 ### Raw diffs
+
 - [capa v3.2.0...v3.2.1](https://github.com/mandiant/capa/compare/v3.2.0...v3.2.1)
 - [capa-rules v3.2.0...v3.2.1](https://github.com/mandiant/capa-rules/compare/v3.2.0...v3.2.1)
 
 ## v3.2.0 (2022-03-03)
+
 This release adds a new characteristic `characteristic: call $+5` enabling users to create more explicit rules. The linter now also validates ATT&CK and MBC categories. Additionally, many dependencies, including the vivisect backend, have been updated.
 
 One rule has been added and many more have been improved.
@@ -687,10 +1100,12 @@ Thanks for all the support, especially to @kn0wl3dge and first time contributor
 - elf: fix OS detection for Linux kernel modules #867 @williballenthin
 
 ### Raw diffs
+
 - [capa v3.1.0...v3.2.0](https://github.com/mandiant/capa/compare/v3.1.0...v3.2.0)
 - [capa-rules v3.1.0...v3.2.0](https://github.com/mandiant/capa-rules/compare/v3.1.0...v3.2.0)
 
 ## v3.1.0 (2022-01-10)
+
 This release improves the performance of capa while also adding 23 new rules and many code quality enhancements. We profiled capa's CPU usage and optimized the way that it matches rules, such as by short circuiting when appropriate. According to our testing, the matching phase is approximately 66% faster than v3.0.3! We also added support for Python 3.10, aarch64 builds, and additional MAEC metadata in the rule headers.
 
 This release adds 23 new rules, including nine by Jakub Jozwiak of Mandiant. @ryantxu1 and @dzbeck updated the ATT&CK and MBC mappings for many rules. Thank you!
@@ -701,7 +1116,6 @@ And as always, welcome first time contributors!
 - @jtothej
 - @cl30
 
-
 ### New Features
 
 - engine: short circuit logic nodes for better performance #824 @williballenthin
@@ -760,23 +1174,24 @@ And as always, welcome first time contributors!
 - show features script: add backend flag #430 @kn0wl3dge
 
 ### Raw diffs
+
 - [capa v3.0.3...v3.1.0](https://github.com/mandiant/capa/compare/v3.0.3...v3.1.0)
 - [capa-rules v3.0.3...v3.1.0](https://github.com/mandiant/capa-rules/compare/v3.0.3...v3.1.0)
 
-
 ## v3.0.3 (2021-10-27)
 
 This is primarily a rule maintenance release:
+
 - eight new rules, including all relevant techniques from [ATT&CK v10](https://medium.com/mitre-attack/introducing-attack-v10-7743870b37e3), and
 - two rules removed, due to the prevalence of false positives
 
 We've also tweaked the status codes returned by capa.exe to be more specific and added a bit more metadata to the JSON output format.
 
 As always, welcome first time contributors!
+
 - still@teamt5.org
 - zander.work@mandiant.com
 
-
 ### New Features
 
 - show in which function a BB match is #130 @williballenthin
@@ -794,6 +1209,7 @@ As always, welcome first time contributors!
 - targeting/language/identify-system-language-via-api william.ballenthin@mandiant.com
 
 ## Removed rules (2)
+
 - load-code/pe/parse-pe-exports: too many false positives in unrelated structure accesses
 - anti-analysis/anti-vm/vm-detection/execute-anti-vm-instructions: too many false positives in junk code
 
@@ -802,6 +1218,7 @@ As always, welcome first time contributors!
 - update references from FireEye to Mandiant
 
 ### Raw diffs
+
 - [capa v3.0.2...v3.0.3](https://github.com/fireeye/capa/compare/v3.0.2...v3.0.3)
 - [capa-rules v3.0.2...v3.0.3](https://github.com/fireeye/capa-rules/compare/v3.0.2...v3.0.3)
 
@@ -814,6 +1231,7 @@ This release fixes an issue with the standalone executables built with PyInstall
 - fix bug in PyInstaller config preventing ELF analysis #795 @mr-tz
 
 ### Raw diffs
+
 - [capa v3.0.1...v3.0.2](https://github.com/fireeye/capa/compare/v3.0.1...v3.0.2)
 - [capa-rules v3.0.1...v3.0.2](https://github.com/fireeye/capa-rules/compare/v3.0.1...v3.0.2)
 
@@ -828,6 +1246,7 @@ Thanks to the community for highlighting issues and analysis misses. Your feedba
 - fix many underlying bugs in vivisect analysis and update to version v1.0.5 #786 @williballenthin
 
 ### Raw diffs
+
 - [capa v3.0.0...v3.0.1](https://github.com/fireeye/capa/compare/v3.0.0...v3.0.1)
 - [capa-rules v3.0.0...v3.0.1](https://github.com/fireeye/capa-rules/compare/v3.0.0...v3.0.1)
 
@@ -836,6 +1255,7 @@ Thanks to the community for highlighting issues and analysis misses. Your feedba
 We are excited to announce version 3.0! :tada:
 
 capa 3.0:
+
 - adds support for ELF files targeting Linux thanks to [Intezer](https://www.intezer.com/)
 - adds new features to specify OS, CPU architecture, and file format
 - fixes a few bugs that may have led to false negatives (missed capabilities) in older versions
@@ -844,6 +1264,7 @@ capa 3.0:
 A huge thanks to everyone who submitted issues, provided feedback, and contributed code and rules.
 Special acknowledgement to @Adir-Shemesh and @TcM1911 of [Intezer](https://www.intezer.com/) for contributing the code to enable ELF support.
 Also, welcome first time contributors:
+
 - @jaredscottwilson
 - @cdong1012
 - @jlepore-fe
@@ -951,14 +1372,15 @@ Also, welcome first time contributors:
 ### Development
 
 ### Raw diffs
+
 - [capa v2.0.0...v3.0.0](https://github.com/mandiant/capa/compare/v2.0.0...v3.0.0)
 - [capa-rules v2.0.0...v3.0.0](https://github.com/mandiant/capa-rules/compare/v2.0.0...v3.0.0)
 
-
 ## v2.0.0 (2021-07-19)
 
 We are excited to announce version 2.0! :tada:
 capa 2.0:
+
 - enables anyone to contribute rules more easily
 - is the first Python 3 ONLY version
 - provides more concise and relevant result via identification of library functions using FLIRT
@@ -968,7 +1390,6 @@ capa 2.0:
 
 A huge thanks to everyone who submitted issues, provided feedback, and contributed code and rules. Many colleagues across dozens of organizations have volunteered their experience to improve this tool! :heart:
 
-
 ### New Features
 
 - rules: update ATT&CK and MBC mappings https://github.com/mandiant/capa-rules/pull/317 @williballenthin
@@ -1095,6 +1516,7 @@ A huge thanks to everyone who submitted issues, provided feedback, and contribut
 - main: do not process non-PE files even when --format explicitly provided #664 @mr-tz
 
 ### capa explorer IDA Pro plugin
+
 - explorer: IDA 7.6 support #497 @williballenthin
 - explorer: explain how to install IDA 7.6 patch to enable the plugin #528 @williballenthin
 - explorer: document IDA 7.6sp1 as alternative to the patch #536 @Ana06
@@ -1115,10 +1537,10 @@ A huge thanks to everyone who submitted issues, provided feedback, and contribut
 ### Raw diffs
 
 <!-- The diff uses v1.6.1 because master doesn't include v1.6.2 and v1.6.3 -->
+
 - [capa v1.6.1...v2.0.0](https://github.com/mandiant/capa/compare/v1.6.1...v2.0.0)
 - [capa-rules v1.6.1...v2.0.0](https://github.com/mandiant/capa-rules/compare/v1.6.1...v2.0.0)
 
-
 ## v1.6.3 (2021-04-29)
 
 This release adds IDA 7.6 support to capa.
@@ -1131,7 +1553,6 @@ This release adds IDA 7.6 support to capa.
 
 - [capa v1.6.2...v1.6.3](https://github.com/mandiant/capa/compare/v1.6.2...v1.6.3)
 
-
 ## v1.6.2 (2021-04-13)
 
 This release backports a fix to capa 1.6: The Windows binary was built with Python 3.9 which doesn't support Windows 7.
@@ -1144,7 +1565,6 @@ This release backports a fix to capa 1.6: The Windows binary was built with Pyth
 
 - [capa v1.6.1...v1.6.2](https://github.com/mandiant/capa/compare/v1.6.1...v1.6.2)
 
-
 ## v1.6.1 (2021-04-07)
 
 This release includes several bug fixes, such as a vivisect issue that prevented capa from working on Windows with Python 3. It also adds 17 new rules and a bunch of improvements in the rules and IDA rule generator. We appreciate everyone who opened issues, provided feedback, and contributed code and rules.
@@ -1213,7 +1633,6 @@ This release includes several bug fixes, such as a vivisect issue that prevented
 - [capa v1.6.0...v1.6.1](https://github.com/mandiant/capa/compare/v1.6.0...v1.6.1)
 - [capa-rules v1.6.0...v1.6.1](https://github.com/mandiant/capa-rules/compare/v1.6.0...v1.6.1)
 
-
 ## v1.6.0 (2021-03-09)
 
 This release adds the capa explorer rule generator plugin for IDA Pro, vivisect support for Python 3 and 12 new rules. We appreciate everyone who opened issues, provided feedback, and contributed code and rules. Thank you also to the vivisect development team (@rakuy0, @atlas0fd00m) for the Python 3 support (`vivisect==1.0.0`) and the fixes for Python 2 (`vivisect==0.2.1`).
@@ -1229,6 +1648,7 @@ The capa explorer IDA plugin now helps you quickly build new capa rules using fe
 This version of capa adds Python 3 support in vivisect. Note that `.viv` files (generated by vivisect) are not compatible between Python 2 and Python 3. When updating to Python 3 you need to delete all the `.viv` files for capa to work.
 
 If you get the following error (or a similar one), you most likely need to delete `.viv` files:
+
 ```
 UnicodeDecodeError: 'ascii' codec can't decode byte 0x90 in position 2: ordinal not in range(128)
 ```
@@ -1285,7 +1705,6 @@ If you have workflows that rely on the Python 2 version and need future maintena
 - [capa v1.5.1...v1.6.0](https://github.com/mandiant/capa/compare/v1.5.1...v1.6.0)
 - [capa-rules v1.5.1...v1.6.0](https://github.com/mandiant/capa-rules/compare/v1.5.1...v1.6.0)
 
-
 ## v1.5.1 (2021-02-09)
 
 This release fixes the version number that we forgot to update for v1.5.0 (therefore, v1.5.0 was not published to pypi). It also includes 1 new rule and some rule improvements.
@@ -1299,7 +1718,6 @@ This release fixes the version number that we forgot to update for v1.5.0 (there
 - [capa v1.5.0...v1.5.1](https://github.com/mandiant/capa/compare/v1.5.1...v1.6.0)
 - [capa-rules v1.5.0...v1.5.1](https://github.com/mandiant/capa-rules/compare/v1.5.1...v1.6.0)
 
-
 ## v1.5.0 (2021-02-05)
 
 This release brings support for running capa under Python 3 via [SMDA](https://github.com/danielplohmann/smda), more thorough CI testing and linting, better extraction of strings and byte features, and 50 (!) new rules. We appreciate everyone who opened issues, provided feedback, and contributed code and rules. A special shout out to the following new project contributors:
@@ -1314,7 +1732,6 @@ This release brings support for running capa under Python 3 via [SMDA](https://g
 
 Download a standalone binary below and checkout the readme [here on GitHub](https://github.com/mandiant/capa/). Report issues on our [issue tracker](https://github.com/mandiant/capa/issues) and contribute new rules at [capa-rules](https://github.com/mandiant/capa-rules/).
 
-
 ### New Features
 
 - py3 support via SMDA #355 @danielplohmann @jcrussell
@@ -1625,6 +2042,7 @@ and contribute new rules at [capa-rules](https://github.com/mandiant/capa-rules/
 - core: pin dependency versions #258 @recvfrom
 
 ### New rules
+
 - bypass UAC via AppInfo ALPC @agithubuserlol
 - bypass UAC via token manipulation @agithubuserlol
 - check for sandbox and av modules @re-fox

LICENSE.txt

@@ -187,7 +187,7 @@
       same "printed page" as the copyright notice for easier
       identification within third-party archives.
 
-   Copyright (C) 2020 Mandiant, Inc.
+   Copyright [yyyy] [name of copyright owner]
 
    Licensed under the Apache License, Version 2.0 (the "License");
    you may not use this file except in compliance with the License.

README.md

@@ -1,4 +1,16 @@
-![capa](https://github.com/mandiant/capa/blob/master/.github/logo.png)
+<br />
+<div align="center">
+<a href="https://mandiant.github.io/capa/" target="_blank">
+  <img src="https://github.com/mandiant/capa/blob/master/.github/logo.png">
+</a>
+<p align="center">
+  <a href="https://mandiant.github.io/capa/" target="_blank">Website</a>
+  |
+  <a href="https://github.com/mandiant/capa/releases/latest" target="_blank">Download</a>
+  |
+  <a href="https://mandiant.github.io/capa/explorer/" target="_blank">Web Interface</a>
+</p>
+<div align="center">
 
 [![PyPI - Python Version](https://img.shields.io/pypi/pyversions/flare-capa)](https://pypi.org/project/flare-capa)
 [![Last release](https://img.shields.io/github/v/release/mandiant/capa)](https://github.com/mandiant/capa/releases)
@@ -7,23 +19,28 @@
 [![Downloads](https://img.shields.io/github/downloads/mandiant/capa/total)](https://github.com/mandiant/capa/releases)
 [![License](https://img.shields.io/badge/license-Apache--2.0-green.svg)](LICENSE.txt)
 
+</div>
+</div>
+
+---
+
 capa detects capabilities in executable files.
 You run it against a PE, ELF, .NET module, shellcode file, or a sandbox report and it tells you what it thinks the program can do.
 For example, it might suggest that the file is a backdoor, is capable of installing services, or relies on HTTP to communicate.
 
-Check out our capa blog posts:
+To interactively inspect capa results in your browser use the [capa Explorer Web](https://mandiant.github.io/capa/explorer/).
-- [Dynamic capa: Exploring Executable Run-Time Behavior with the CAPE Sandbox](https://www.mandiant.com/resources/blog/dynamic-capa-executable-behavior-cape-sandbox)
-- [capa v4: casting a wider .NET](https://www.mandiant.com/resources/blog/capa-v4-casting-wider-net) (.NET support)
-- [ELFant in the Room – capa v3](https://www.mandiant.com/resources/elfant-in-the-room-capa-v3) (ELF support)
-- [capa 2.0: Better, Stronger, Faster](https://www.mandiant.com/resources/capa-2-better-stronger-faster)
-- [capa: Automatically Identify Malware Capabilities](https://www.mandiant.com/resources/capa-automatically-identify-malware-capabilities)
 
+If you want to inspect or write capa rules, head on over to the [capa-rules repository](https://github.com/mandiant/capa-rules). Otherwise, keep reading.
+
+Below you find a list of [our capa blog posts with more details.](#blog-posts)
+
+# example capa output
 ```
 $ capa.exe suspicious.exe
 
-+------------------------+--------------------------------------------------------------------------------+
++--------------------+------------------------------------------------------------------------+
 | ATT&CK Tactic      | ATT&CK Technique                                                       |
-|------------------------+--------------------------------------------------------------------------------|
+|--------------------+------------------------------------------------------------------------|
 | DEFENSE EVASION    | Obfuscated Files or Information [T1027]                                |
 | DISCOVERY          | Query Registry [T1012]                                                 |
 |                    | System Information Discovery [T1082]                                   |
@@ -31,12 +48,11 @@ $ capa.exe suspicious.exe
 |                    | Shared Modules [T1129]                                                 |
 | EXFILTRATION       | Exfiltration Over C2 Channel [T1041]                                   |
 | PERSISTENCE        | Create or Modify System Process::Windows Service [T1543.003]           |
-+------------------------+--------------------------------------------------------------------------------+
++--------------------+------------------------------------------------------------------------+
 
-+-------------------------------------------------------+-------------------------------------------------+
++-------------------------------------------+-------------------------------------------------+
 | CAPABILITY                                | NAMESPACE                                       |
-|-------------------------------------------------------+-------------------------------------------------|
+|-------------------------------------------+-------------------------------------------------|
-| check for OutputDebugString error                     | anti-analysis/anti-debugging/debugger-detection |
 | read and send data from client to server  | c2/file-transfer                               |
 | execute shell command and capture output  | c2/shell                                       |
 | receive data (2 matches)                  | communication                                   |
@@ -57,13 +73,12 @@ $ capa.exe suspicious.exe
 | print debug messages (2 matches)          | host-interaction/log/debug/write-event          |
 | resolve DNS                               | host-interaction/network/dns/resolve            |
 | get hostname                              | host-interaction/os/hostname                    |
-| create a process with modified I/O handles and window | host-interaction/process/create                 |
 | create process                            | host-interaction/process/create                 |
 | create registry key                       | host-interaction/registry/create                |
 | create service                            | host-interaction/service/create                 |
 | create thread                             | host-interaction/thread/create                  |
 | persist via Windows service               | persistence/service                             |
-+-------------------------------------------------------+-------------------------------------------------+
++-------------------------------------------+-------------------------------------------------+
 ```
 
 # download and usage
@@ -72,16 +87,23 @@ Download stable releases of the standalone capa binaries [here](https://github.c
 
 To use capa as a library or integrate with another tool, see [doc/installation.md](https://github.com/mandiant/capa/blob/master/doc/installation.md) for further setup instructions.
 
-For more information about how to use capa, see [doc/usage.md](https://github.com/mandiant/capa/blob/master/doc/usage.md).
+# capa Explorer Web
+The [capa Explorer Web](https://mandiant.github.io/capa/explorer/) enables you to interactively explore capa results in your web browser. Besides the online version you can download a standalone HTML file for local offline usage.
+
+![capa Explorer Web screenshot](https://github.com/mandiant/capa/blob/master/doc/img/capa_web_explorer.png)
+
+More details on the web UI is available in the [capa Explorer Web README](https://github.com/mandiant/capa/blob/master/web/explorer/README.md).
 
 # example
 
-In the above sample output, we ran capa against an unknown binary (`suspicious.exe`),
+In the above sample output, we run capa against an unknown binary (`suspicious.exe`),
-and the tool reported that the program can send HTTP requests, decode data via XOR and Base64,
+and the tool reports that the program can send HTTP requests, decode data via XOR and Base64,
 install services, and spawn new processes.
 Taken together, this makes us think that `suspicious.exe` could be a persistent backdoor.
 Therefore, our next analysis step might be to run `suspicious.exe` in a sandbox and try to recover the command and control server.
 
+## detailed results
+
 By passing the `-vv` flag (for very verbose), capa reports exactly where it found evidence of these capabilities.
 This is useful for at least two reasons:
 
@@ -126,10 +148,15 @@ function @ 0x4011C0
 ...
 ```
 
-Additionally, capa also supports analyzing [CAPE](https://github.com/kevoreilly/CAPEv2) sandbox reports for dynamic capability extraction.
+capa also supports dynamic capabilities detection for multiple sandboxes including:
-In order to use this, you first submit your sample to CAPE for analysis, and then run capa against the generated report (JSON).
+* [CAPE](https://github.com/kevoreilly/CAPEv2) (supported report formats: `.json`, `.json_`, `.json.gz`)
+* [DRAKVUF](https://github.com/CERT-Polska/drakvuf-sandbox/) (supported report formats: `.log`, `.log.gz`)
+* [VMRay](https://www.vmray.com/) (supported report formats: analysis archive `.zip`)
 
-Here's an example of running capa against a packed binary, and then running capa against the CAPE report of that binary:
+
+To use this feature, submit your file to a supported sandbox and then download and run capa against the generated report file. This feature enables capa to match capabilities against dynamic and static features that the sandbox captured during execution.
+
+Here's an example of running capa against a packed file, and then running capa against the CAPE report generated for the same packed file:
 
 ```yaml
 $ capa 05be49819139a3fdcdbddbdefd298398779521f3d68daa25275cc77508e42310.exe
@@ -216,6 +243,7 @@ $ capa 05be49819139a3fdcdbddbdefd298398779521f3d68daa25275cc77508e42310.json
 ┕━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┙
 ```
 
+# capa rules
 capa uses a collection of rules to identify capabilities within a program.
 These rules are easy to write, even for those new to reverse engineering.
 By authoring rules, you can extend the capabilities that capa recognizes.
@@ -252,18 +280,28 @@ rule:
       - property/read: System.Net.Sockets.TcpClient::Client
 ```
 
-The [github.com/mandiant/capa-rules](https://github.com/mandiant/capa-rules) repository contains hundreds of standard library rules that are distributed with capa.
+The [github.com/mandiant/capa-rules](https://github.com/mandiant/capa-rules) repository contains hundreds of standard rules that are distributed with capa.
 Please learn to write rules and contribute new entries as you find interesting techniques in malware.
 
+# IDA Pro plugin: capa explorer
 If you use IDA Pro, then you can use the [capa explorer](https://github.com/mandiant/capa/tree/master/capa/ida/plugin) plugin.
 capa explorer helps you identify interesting areas of a program and build new capa rules using features extracted directly from your IDA Pro database.
+It also uses your local changes to the .idb to extract better features, such as when you rename a global variable that contains a dynamically resolved API address.
 
 ![capa + IDA Pro integration](https://github.com/mandiant/capa/blob/master/doc/img/explorer_expanded.png)
 
+# Ghidra integration
 If you use Ghidra, then you can use the [capa + Ghidra integration](/capa/ghidra/) to run capa's analysis directly on your Ghidra database and render the results in Ghidra's user interface.
 
 <img src="https://github.com/mandiant/capa/assets/66766340/eeae33f4-99d4-42dc-a5e8-4c1b8c661492" width=300>
 
+# blog posts
+- [Dynamic capa: Exploring Executable Run-Time Behavior with the CAPE Sandbox](https://www.mandiant.com/resources/blog/dynamic-capa-executable-behavior-cape-sandbox)
+- [capa v4: casting a wider .NET](https://www.mandiant.com/resources/blog/capa-v4-casting-wider-net) (.NET support)
+- [ELFant in the Room – capa v3](https://www.mandiant.com/resources/elfant-in-the-room-capa-v3) (ELF support)
+- [capa 2.0: Better, Stronger, Faster](https://www.mandiant.com/resources/capa-2-better-stronger-faster)
+- [capa: Automatically Identify Malware Capabilities](https://www.mandiant.com/resources/capa-automatically-identify-malware-capabilities)
+
 # further information
 ## capa
 - [Installation](https://github.com/mandiant/capa/blob/master/doc/installation.md)

capa/capabilities/common.py

@@ -1,25 +1,43 @@
 # -*- coding: utf-8 -*-
-# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
+# Copyright 2023 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# You may obtain a copy of the License at
-# Unless required by applicable law or agreed to in writing, software distributed under the License
+#
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#     http://www.apache.org/licenses/LICENSE-2.0
-# See the License for the specific language governing permissions and limitations under the License.
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 import logging
 import itertools
 import collections
-from typing import Any, Tuple
+from typing import Optional
+from dataclasses import dataclass
 
-from capa.rules import Scope, RuleSet
+from capa.rules import Rule, Scope, RuleSet
 from capa.engine import FeatureSet, MatchResults
 from capa.features.address import NO_ADDRESS
+from capa.render.result_document import LibraryFunction, StaticFeatureCounts, DynamicFeatureCounts
 from capa.features.extractors.base_extractor import FeatureExtractor, StaticFeatureExtractor, DynamicFeatureExtractor
 
 logger = logging.getLogger(__name__)
 
 
-def find_file_capabilities(ruleset: RuleSet, extractor: FeatureExtractor, function_features: FeatureSet):
+@dataclass
+class FileCapabilities:
+    features: FeatureSet
+    matches: MatchResults
+    feature_count: int
+
+
+def find_file_capabilities(
+    ruleset: RuleSet, extractor: FeatureExtractor, function_features: FeatureSet
+) -> FileCapabilities:
     file_features: FeatureSet = collections.defaultdict(set)
 
     for feature, va in itertools.chain(extractor.extract_file_features(), extractor.extract_global_features()):
@@ -36,35 +54,18 @@ def find_file_capabilities(ruleset: RuleSet, extractor: FeatureExtractor, functi
 
     file_features.update(function_features)
 
-    _, matches = ruleset.match(Scope.FILE, file_features, NO_ADDRESS)
+    features, matches = ruleset.match(Scope.FILE, file_features, NO_ADDRESS)
-    return matches, len(file_features)
+    return FileCapabilities(features, matches, len(file_features))
 
 
-def has_file_limitation(rules: RuleSet, capabilities: MatchResults, is_standalone=True) -> bool:
+@dataclass
-    file_limitation_rules = list(filter(lambda r: r.is_file_limitation_rule(), rules.rules.values()))
+class Capabilities:
-
+    matches: MatchResults
-    for file_limitation_rule in file_limitation_rules:
+    feature_counts: StaticFeatureCounts | DynamicFeatureCounts
-        if file_limitation_rule.name not in capabilities:
+    library_functions: Optional[tuple[LibraryFunction, ...]] = None
-            continue
-
-        logger.warning("-" * 80)
-        for line in file_limitation_rule.meta.get("description", "").split("\n"):
-            logger.warning(" %s", line)
-        logger.warning(" Identified via rule: %s", file_limitation_rule.name)
-        if is_standalone:
-            logger.warning(" ")
-            logger.warning(" Use -v or -vv if you really want to see the capabilities identified by capa.")
-        logger.warning("-" * 80)
-
-        # bail on first file limitation
-        return True
-
-    return False
 
 
-def find_capabilities(
+def find_capabilities(ruleset: RuleSet, extractor: FeatureExtractor, disable_progress=None, **kwargs) -> Capabilities:
-    ruleset: RuleSet, extractor: FeatureExtractor, disable_progress=None, **kwargs
-) -> Tuple[MatchResults, Any]:
     from capa.capabilities.static import find_static_capabilities
     from capa.capabilities.dynamic import find_dynamic_capabilities
 
@@ -77,3 +78,40 @@ def find_capabilities(
         return find_dynamic_capabilities(ruleset, extractor, disable_progress=disable_progress, **kwargs)
 
     raise ValueError(f"unexpected extractor type: {extractor.__class__.__name__}")
+
+
+def has_limitation(rules: list, capabilities: Capabilities | FileCapabilities, is_standalone: bool) -> bool:
+
+    for rule in rules:
+        if rule.name not in capabilities.matches:
+            continue
+        logger.warning("-" * 80)
+        for line in rule.meta.get("description", "").split("\n"):
+            logger.warning(" %s", line)
+        logger.warning(" Identified via rule: %s", rule.name)
+        if is_standalone:
+            logger.warning(" ")
+            logger.warning(" Use -v or -vv if you really want to see the capabilities identified by capa.")
+        logger.warning("-" * 80)
+
+        # bail on first file limitation
+        return True
+    return False
+
+
+def is_static_limitation_rule(r: Rule) -> bool:
+    return r.meta.get("namespace", "") == "internal/limitation/static"
+
+
+def has_static_limitation(rules: RuleSet, capabilities: Capabilities | FileCapabilities, is_standalone=True) -> bool:
+    file_limitation_rules = list(filter(lambda r: is_static_limitation_rule(r), rules.rules.values()))
+    return has_limitation(file_limitation_rules, capabilities, is_standalone)
+
+
+def is_dynamic_limitation_rule(r: Rule) -> bool:
+    return r.meta.get("namespace", "") == "internal/limitation/dynamic"
+
+
+def has_dynamic_limitation(rules: RuleSet, capabilities: Capabilities | FileCapabilities, is_standalone=True) -> bool:
+    dynamic_limitation_rules = list(filter(lambda r: is_dynamic_limitation_rule(r), rules.rules.values()))
+    return has_limitation(dynamic_limitation_rules, capabilities, is_standalone)

capa/capabilities/dynamic.py

@@ -1,38 +1,55 @@
 # -*- coding: utf-8 -*-
-# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
+# Copyright 2023 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# You may obtain a copy of the License at
-# Unless required by applicable law or agreed to in writing, software distributed under the License
+#
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#     http://www.apache.org/licenses/LICENSE-2.0
-# See the License for the specific language governing permissions and limitations under the License.
+#
-import sys
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 import logging
 import itertools
 import collections
-from typing import Any, Tuple
+from dataclasses import dataclass
-
-import tqdm
 
 import capa.perf
+import capa.engine
+import capa.helpers
 import capa.features.freeze as frz
 import capa.render.result_document as rdoc
 from capa.rules import Scope, RuleSet
 from capa.engine import FeatureSet, MatchResults
-from capa.helpers import redirecting_print_to_tqdm
+from capa.features.address import _NoAddress
-from capa.capabilities.common import find_file_capabilities
+from capa.capabilities.common import Capabilities, find_file_capabilities
 from capa.features.extractors.base_extractor import CallHandle, ThreadHandle, ProcessHandle, DynamicFeatureExtractor
 
 logger = logging.getLogger(__name__)
 
 
+# The number of calls that make up a span of calls.
+#
+# The larger this is, the more calls are grouped together to match rule logic.
+# This means a longer chain can be recognized; however, its a bit more expensive.
+SPAN_SIZE = 20
+
+
+@dataclass
+class CallCapabilities:
+    features: FeatureSet
+    matches: MatchResults
+
+
 def find_call_capabilities(
     ruleset: RuleSet, extractor: DynamicFeatureExtractor, ph: ProcessHandle, th: ThreadHandle, ch: CallHandle
-) -> Tuple[FeatureSet, MatchResults]:
+) -> CallCapabilities:
     """
     find matches for the given rules for the given call.
-
-    returns: tuple containing (features for call, match results for call)
     """
     # all features found for the call.
     features: FeatureSet = collections.defaultdict(set)
@@ -50,16 +67,105 @@ def find_call_capabilities(
         for addr, _ in res:
             capa.engine.index_rule_matches(features, rule, [addr])
 
-    return features, matches
+    return CallCapabilities(features, matches)
+
+
+@dataclass
+class ThreadCapabilities:
+    features: FeatureSet
+    thread_matches: MatchResults
+    span_matches: MatchResults
+    call_matches: MatchResults
+
+
+class SpanOfCallsMatcher:
+    def __init__(self, ruleset: RuleSet):
+        super().__init__()
+        self.ruleset = ruleset
+
+        # matches found at the span scope.
+        self.matches: MatchResults = collections.defaultdict(list)
+
+        # We match spans as the sliding window of calls with size SPAN_SIZE.
+        #
+        # For each call, we consider the window of SPAN_SIZE calls leading up to it,
+        #  merging all their features and doing a match.
+        #
+        # We track these features in two data structures:
+        #   1. a deque of those features found in the prior calls.
+        #      We'll append to it, and as it grows larger than SPAN_SIZE, the oldest items are removed.
+        #   2. a live set of features seen in the span.
+        #      As we pop from the deque, we remove features from the current set,
+        #      and as we push to the deque, we insert features to the current set.
+        # With this approach, our algorithm performance is independent of SPAN_SIZE.
+        # The naive algorithm, of merging all the trailing feature sets at each call, is dependent upon SPAN_SIZE
+        # (that is, runtime gets slower the larger SPAN_SIZE is).
+        self.current_feature_sets: collections.deque[FeatureSet] = collections.deque(maxlen=SPAN_SIZE)
+        self.current_features: FeatureSet = collections.defaultdict(set)
+
+        # the names of rules matched at the last span,
+        # so that we can deduplicate long strings of the same matches.
+        self.last_span_matches: set[str] = set()
+
+    def next(self, ch: CallHandle, call_features: FeatureSet):
+        # As we add items to the end of the deque, overflow and drop the oldest items (at the left end).
+        # While we could rely on `deque.append` with `maxlen` set (which we provide above),
+        # we want to use the dropped item first, to remove the old features, so we manually pop it here.
+        if len(self.current_feature_sets) == SPAN_SIZE:
+            overflowing_feature_set = self.current_feature_sets.popleft()
+
+            for feature, vas in overflowing_feature_set.items():
+                if len(vas) == 1 and isinstance(next(iter(vas)), _NoAddress):
+                    # `vas == { NO_ADDRESS }` without the garbage.
+                    #
+                    # ignore the common case of global features getting added/removed/trimmed repeatedly,
+                    # like arch/os/format.
+                    continue
+
+                self.current_features[feature] -= vas
+                if not self.current_features[feature]:
+                    del self.current_features[feature]
+
+        # update the deque and set of features with the latest call's worth of features.
+        self.current_feature_sets.append(call_features)
+        for feature, vas in call_features.items():
+            self.current_features[feature] |= vas
+
+        _, matches = self.ruleset.match(Scope.SPAN_OF_CALLS, self.current_features, ch.address)
+
+        newly_encountered_rules = set(matches.keys()) - self.last_span_matches
+
+        # don't emit match results for rules seen during the immediately preceeding spans.
+        #
+        # This means that we won't emit duplicate matches when there are multiple spans
+        #  that overlap a single matching event.
+        # It also handles the case of a tight loop containing matched logic;
+        #  only the first match will be recorded.
+        #
+        # In theory, this means the result document doesn't have *every* possible match location,
+        # but in practice, humans will only be interested in the first handful anyways.
+        suppressed_rules = set(self.last_span_matches)
+
+        # however, if a newly encountered rule depends on a suppressed rule,
+        # don't suppress that rule match, or we won't be able to reconstruct the vverbose output.
+        # see: https://github.com/mandiant/capa/pull/2532#issuecomment-2548508130
+        for new_rule in newly_encountered_rules:
+            suppressed_rules -= set(self.ruleset.rules[new_rule].get_dependencies(self.ruleset.rules_by_namespace))
+
+        for rule_name, res in matches.items():
+            if rule_name in suppressed_rules:
+                continue
+            self.matches[rule_name].extend(res)
+
+        self.last_span_matches = set(matches.keys())
 
 
 def find_thread_capabilities(
     ruleset: RuleSet, extractor: DynamicFeatureExtractor, ph: ProcessHandle, th: ThreadHandle
-) -> Tuple[FeatureSet, MatchResults, MatchResults]:
+) -> ThreadCapabilities:
     """
-    find matches for the given rules within the given thread.
+    find matches for the given rules within the given thread,
-
+    which includes matches for all the spans and calls within it.
-    returns: tuple containing (features for thread, match results for thread, match results for calls)
     """
     # all features found within this thread,
     # includes features found within calls.
@@ -69,14 +175,19 @@ def find_thread_capabilities(
     # might be found at different calls, that's ok.
     call_matches: MatchResults = collections.defaultdict(list)
 
-    for ch in extractor.get_calls(ph, th):
+    span_matcher = SpanOfCallsMatcher(ruleset)
-        ifeatures, imatches = find_call_capabilities(ruleset, extractor, ph, th, ch)
+
-        for feature, vas in ifeatures.items():
+    call_count = 0
+    for call_count, ch in enumerate(extractor.get_calls(ph, th)):  # noqa: B007
+        call_capabilities = find_call_capabilities(ruleset, extractor, ph, th, ch)
+        for feature, vas in call_capabilities.features.items():
             features[feature].update(vas)
 
-        for rule_name, res in imatches.items():
+        for rule_name, res in call_capabilities.matches.items():
             call_matches[rule_name].extend(res)
 
+        span_matcher.next(ch, call_capabilities.features)
+
     for feature, va in itertools.chain(extractor.extract_thread_features(ph, th), extractor.extract_global_features()):
         features[feature].add(va)
 
@@ -88,16 +199,31 @@ def find_thread_capabilities(
         for va, _ in res:
             capa.engine.index_rule_matches(features, rule, [va])
 
-    return features, matches, call_matches
+    logger.debug(
+        "analyzed thread %d[%d] with %d events, %d features, and %d matches",
+        th.address.process.pid,
+        th.address.tid,
+        call_count,
+        len(features),
+        len(matches) + len(span_matcher.matches) + len(call_matches),
+    )
+    return ThreadCapabilities(features, matches, span_matcher.matches, call_matches)
+
+
+@dataclass
+class ProcessCapabilities:
+    process_matches: MatchResults
+    thread_matches: MatchResults
+    span_matches: MatchResults
+    call_matches: MatchResults
+    feature_count: int
 
 
 def find_process_capabilities(
     ruleset: RuleSet, extractor: DynamicFeatureExtractor, ph: ProcessHandle
-) -> Tuple[MatchResults, MatchResults, MatchResults, int]:
+) -> ProcessCapabilities:
     """
     find matches for the given rules within the given process.
-
-    returns: tuple containing (match results for process, match results for threads, match results for calls, number of features)
     """
     # all features found within this process,
     # includes features found within threads (and calls).
@@ -107,98 +233,103 @@ def find_process_capabilities(
     # might be found at different threads, that's ok.
     thread_matches: MatchResults = collections.defaultdict(list)
 
+    # matches found at the span-of-calls scope.
+    # might be found at different spans, that's ok.
+    span_matches: MatchResults = collections.defaultdict(list)
+
     # matches found at the call scope.
     # might be found at different calls, that's ok.
     call_matches: MatchResults = collections.defaultdict(list)
 
     for th in extractor.get_threads(ph):
-        features, tmatches, cmatches = find_thread_capabilities(ruleset, extractor, ph, th)
+        thread_capabilities = find_thread_capabilities(ruleset, extractor, ph, th)
-        for feature, vas in features.items():
+        for feature, vas in thread_capabilities.features.items():
             process_features[feature].update(vas)
 
-        for rule_name, res in tmatches.items():
+        for rule_name, res in thread_capabilities.thread_matches.items():
             thread_matches[rule_name].extend(res)
 
-        for rule_name, res in cmatches.items():
+        for rule_name, res in thread_capabilities.span_matches.items():
+            span_matches[rule_name].extend(res)
+
+        for rule_name, res in thread_capabilities.call_matches.items():
             call_matches[rule_name].extend(res)
 
     for feature, va in itertools.chain(extractor.extract_process_features(ph), extractor.extract_global_features()):
         process_features[feature].add(va)
 
     _, process_matches = ruleset.match(Scope.PROCESS, process_features, ph.address)
-    return process_matches, thread_matches, call_matches, len(process_features)
+
+    logger.debug(
+        "analyzed process %d and extracted %d features with %d matches",
+        ph.address.pid,
+        len(process_features),
+        len(process_matches),
+    )
+    return ProcessCapabilities(process_matches, thread_matches, span_matches, call_matches, len(process_features))
 
 
 def find_dynamic_capabilities(
-    ruleset: RuleSet, extractor: DynamicFeatureExtractor, disable_progress=None
+    ruleset: RuleSet, extractor: DynamicFeatureExtractor, disable_progress: bool = False
-) -> Tuple[MatchResults, Any]:
+) -> Capabilities:
     all_process_matches: MatchResults = collections.defaultdict(list)
     all_thread_matches: MatchResults = collections.defaultdict(list)
+    all_span_matches: MatchResults = collections.defaultdict(list)
     all_call_matches: MatchResults = collections.defaultdict(list)
 
     feature_counts = rdoc.DynamicFeatureCounts(file=0, processes=())
 
     assert isinstance(extractor, DynamicFeatureExtractor)
-    with redirecting_print_to_tqdm(disable_progress):
+    processes: list[ProcessHandle] = list(extractor.get_processes())
-        with tqdm.contrib.logging.logging_redirect_tqdm():
+    n_processes: int = len(processes)
-            pbar = tqdm.tqdm
-            if disable_progress:
-                # do not use tqdm to avoid unnecessary side effects when caller intends
-                # to disable progress completely
-                def pbar(s, *args, **kwargs):
-                    return s
 
-            elif not sys.stderr.isatty():
+    with capa.helpers.CapaProgressBar(
-                # don't display progress bar when stderr is redirected to a file
+        console=capa.helpers.log_console, transient=True, disable=disable_progress
-                def pbar(s, *args, **kwargs):
+    ) as pbar:
-                    return s
+        task = pbar.add_task("matching", total=n_processes, unit="processes")
-
+        for p in processes:
-            processes = list(extractor.get_processes())
+            process_capabilities = find_process_capabilities(ruleset, extractor, p)
-
-            pb = pbar(processes, desc="matching", unit=" processes", leave=False)
-            for p in pb:
-                process_matches, thread_matches, call_matches, feature_count = find_process_capabilities(
-                    ruleset, extractor, p
-                )
             feature_counts.processes += (
-                    rdoc.ProcessFeatureCount(address=frz.Address.from_capa(p.address), count=feature_count),
+                rdoc.ProcessFeatureCount(
+                    address=frz.Address.from_capa(p.address), count=process_capabilities.feature_count
+                ),
             )
-                logger.debug("analyzed %s and extracted %d features", p.address, feature_count)
 
-                for rule_name, res in process_matches.items():
+            for rule_name, res in process_capabilities.process_matches.items():
                 all_process_matches[rule_name].extend(res)
-                for rule_name, res in thread_matches.items():
+            for rule_name, res in process_capabilities.thread_matches.items():
                 all_thread_matches[rule_name].extend(res)
-                for rule_name, res in call_matches.items():
+            for rule_name, res in process_capabilities.span_matches.items():
+                all_span_matches[rule_name].extend(res)
+            for rule_name, res in process_capabilities.call_matches.items():
                 all_call_matches[rule_name].extend(res)
 
+            pbar.advance(task)
+
     # collection of features that captures the rule matches within process and thread scopes.
     # mapping from feature (matched rule) to set of addresses at which it matched.
     process_and_lower_features: FeatureSet = collections.defaultdict(set)
     for rule_name, results in itertools.chain(
-        all_process_matches.items(), all_thread_matches.items(), all_call_matches.items()
+        all_process_matches.items(), all_thread_matches.items(), all_span_matches.items(), all_call_matches.items()
     ):
         locations = {p[0] for p in results}
         rule = ruleset[rule_name]
         capa.engine.index_rule_matches(process_and_lower_features, rule, locations)
 
-    all_file_matches, feature_count = find_file_capabilities(ruleset, extractor, process_and_lower_features)
+    all_file_capabilities = find_file_capabilities(ruleset, extractor, process_and_lower_features)
-    feature_counts.file = feature_count
+    feature_counts.file = all_file_capabilities.feature_count
 
     matches = dict(
         itertools.chain(
             # each rule exists in exactly one scope,
             # so there won't be any overlap among these following MatchResults,
             # and we can merge the dictionaries naively.
+            all_call_matches.items(),
+            all_span_matches.items(),
             all_thread_matches.items(),
             all_process_matches.items(),
-            all_call_matches.items(),
+            all_file_capabilities.matches.items(),
-            all_file_matches.items(),
         )
     )
 
-    meta = {
+    return Capabilities(matches, feature_counts)
-        "feature_counts": feature_counts,
-    }
-
-    return matches, meta

capa/capabilities/static.py

@@ -1,39 +1,47 @@
 # -*- coding: utf-8 -*-
-# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
+# Copyright 2023 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# You may obtain a copy of the License at
-# Unless required by applicable law or agreed to in writing, software distributed under the License
+#
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#     http://www.apache.org/licenses/LICENSE-2.0
-# See the License for the specific language governing permissions and limitations under the License.
+#
-import sys
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 import time
 import logging
 import itertools
 import collections
-from typing import Any, Tuple
+from dataclasses import dataclass
-
-import tqdm.contrib.logging
 
 import capa.perf
+import capa.helpers
 import capa.features.freeze as frz
 import capa.render.result_document as rdoc
 from capa.rules import Scope, RuleSet
 from capa.engine import FeatureSet, MatchResults
-from capa.helpers import redirecting_print_to_tqdm
+from capa.capabilities.common import Capabilities, find_file_capabilities
-from capa.capabilities.common import find_file_capabilities
 from capa.features.extractors.base_extractor import BBHandle, InsnHandle, FunctionHandle, StaticFeatureExtractor
 
 logger = logging.getLogger(__name__)
 
 
+@dataclass
+class InstructionCapabilities:
+    features: FeatureSet
+    matches: MatchResults
+
+
 def find_instruction_capabilities(
     ruleset: RuleSet, extractor: StaticFeatureExtractor, f: FunctionHandle, bb: BBHandle, insn: InsnHandle
-) -> Tuple[FeatureSet, MatchResults]:
+) -> InstructionCapabilities:
     """
     find matches for the given rules for the given instruction.
-
-    returns: tuple containing (features for instruction, match results for instruction)
     """
     # all features found for the instruction.
     features: FeatureSet = collections.defaultdict(set)
@@ -51,16 +59,21 @@ def find_instruction_capabilities(
         for addr, _ in res:
             capa.engine.index_rule_matches(features, rule, [addr])
 
-    return features, matches
+    return InstructionCapabilities(features, matches)
+
+
+@dataclass
+class BasicBlockCapabilities:
+    features: FeatureSet
+    basic_block_matches: MatchResults
+    instruction_matches: MatchResults
 
 
 def find_basic_block_capabilities(
     ruleset: RuleSet, extractor: StaticFeatureExtractor, f: FunctionHandle, bb: BBHandle
-) -> Tuple[FeatureSet, MatchResults, MatchResults]:
+) -> BasicBlockCapabilities:
     """
     find matches for the given rules within the given basic block.
-
-    returns: tuple containing (features for basic block, match results for basic block, match results for instructions)
     """
     # all features found within this basic block,
     # includes features found within instructions.
@@ -71,11 +84,11 @@ def find_basic_block_capabilities(
     insn_matches: MatchResults = collections.defaultdict(list)
 
     for insn in extractor.get_instructions(f, bb):
-        ifeatures, imatches = find_instruction_capabilities(ruleset, extractor, f, bb, insn)
+        instruction_capabilities = find_instruction_capabilities(ruleset, extractor, f, bb, insn)
-        for feature, vas in ifeatures.items():
+        for feature, vas in instruction_capabilities.features.items():
             features[feature].update(vas)
 
-        for rule_name, res in imatches.items():
+        for rule_name, res in instruction_capabilities.matches.items():
             insn_matches[rule_name].extend(res)
 
     for feature, va in itertools.chain(
@@ -91,16 +104,20 @@ def find_basic_block_capabilities(
         for va, _ in res:
             capa.engine.index_rule_matches(features, rule, [va])
 
-    return features, matches, insn_matches
+    return BasicBlockCapabilities(features, matches, insn_matches)
 
 
-def find_code_capabilities(
+@dataclass
-    ruleset: RuleSet, extractor: StaticFeatureExtractor, fh: FunctionHandle
+class CodeCapabilities:
-) -> Tuple[MatchResults, MatchResults, MatchResults, int]:
+    function_matches: MatchResults
+    basic_block_matches: MatchResults
+    instruction_matches: MatchResults
+    feature_count: int
+
+
+def find_code_capabilities(ruleset: RuleSet, extractor: StaticFeatureExtractor, fh: FunctionHandle) -> CodeCapabilities:
     """
     find matches for the given rules within the given function.
-
-    returns: tuple containing (match results for function, match results for basic blocks, match results for instructions, number of features)
     """
     # all features found within this function,
     # includes features found within basic blocks (and instructions).
@@ -115,58 +132,46 @@ def find_code_capabilities(
     insn_matches: MatchResults = collections.defaultdict(list)
 
     for bb in extractor.get_basic_blocks(fh):
-        features, bmatches, imatches = find_basic_block_capabilities(ruleset, extractor, fh, bb)
+        basic_block_capabilities = find_basic_block_capabilities(ruleset, extractor, fh, bb)
-        for feature, vas in features.items():
+        for feature, vas in basic_block_capabilities.features.items():
             function_features[feature].update(vas)
 
-        for rule_name, res in bmatches.items():
+        for rule_name, res in basic_block_capabilities.basic_block_matches.items():
             bb_matches[rule_name].extend(res)
 
-        for rule_name, res in imatches.items():
+        for rule_name, res in basic_block_capabilities.instruction_matches.items():
             insn_matches[rule_name].extend(res)
 
     for feature, va in itertools.chain(extractor.extract_function_features(fh), extractor.extract_global_features()):
         function_features[feature].add(va)
 
     _, function_matches = ruleset.match(Scope.FUNCTION, function_features, fh.address)
-    return function_matches, bb_matches, insn_matches, len(function_features)
+    return CodeCapabilities(function_matches, bb_matches, insn_matches, len(function_features))
 
 
 def find_static_capabilities(
     ruleset: RuleSet, extractor: StaticFeatureExtractor, disable_progress=None
-) -> Tuple[MatchResults, Any]:
+) -> Capabilities:
     all_function_matches: MatchResults = collections.defaultdict(list)
     all_bb_matches: MatchResults = collections.defaultdict(list)
     all_insn_matches: MatchResults = collections.defaultdict(list)
 
     feature_counts = rdoc.StaticFeatureCounts(file=0, functions=())
-    library_functions: Tuple[rdoc.LibraryFunction, ...] = ()
+    library_functions: tuple[rdoc.LibraryFunction, ...] = ()
 
     assert isinstance(extractor, StaticFeatureExtractor)
-    with redirecting_print_to_tqdm(disable_progress):
+    functions: list[FunctionHandle] = list(extractor.get_functions())
-        with tqdm.contrib.logging.logging_redirect_tqdm():
+    n_funcs: int = len(functions)
-            pbar = tqdm.tqdm
+    n_libs: int = 0
-            if capa.helpers.is_runtime_ghidra():
+    percentage: float = 0
-                # Ghidrathon interpreter cannot properly handle
-                # the TMonitor thread that is created via a monitor_interval
-                # > 0
-                pbar.monitor_interval = 0
-            if disable_progress:
-                # do not use tqdm to avoid unnecessary side effects when caller intends
-                # to disable progress completely
-                def pbar(s, *args, **kwargs):
-                    return s
 
-            elif not sys.stderr.isatty():
+    with capa.helpers.CapaProgressBar(
-                # don't display progress bar when stderr is redirected to a file
+        console=capa.helpers.log_console, transient=True, disable=disable_progress
-                def pbar(s, *args, **kwargs):
+    ) as pbar:
-                    return s
+        task = pbar.add_task(
-
+            "matching", total=n_funcs, unit="functions", postfix=f"skipped {n_libs} library functions, {percentage}%"
-            functions = list(extractor.get_functions())
+        )
-            n_funcs = len(functions)
+        for f in functions:
-
-            pb = pbar(functions, desc="matching", unit=" functions", postfix="skipped 0 library functions", leave=False)
-            for f in pb:
             t0 = time.time()
             if extractor.is_library_function(f.address):
                 function_name = extractor.get_function_name(f.address)
@@ -176,43 +181,44 @@ def find_static_capabilities(
                 )
                 n_libs = len(library_functions)
                 percentage = round(100 * (n_libs / n_funcs))
-                    if isinstance(pb, tqdm.tqdm):
+                pbar.update(task, postfix=f"skipped {n_libs} library functions, {percentage}%")
-                        pb.set_postfix_str(f"skipped {n_libs} library functions ({percentage}%)")
+                pbar.advance(task)
                 continue
 
-                function_matches, bb_matches, insn_matches, feature_count = find_code_capabilities(
+            code_capabilities = find_code_capabilities(ruleset, extractor, f)
-                    ruleset, extractor, f
-                )
             feature_counts.functions += (
-                    rdoc.FunctionFeatureCount(address=frz.Address.from_capa(f.address), count=feature_count),
+                rdoc.FunctionFeatureCount(
+                    address=frz.Address.from_capa(f.address), count=code_capabilities.feature_count
+                ),
             )
             t1 = time.time()
 
             match_count = 0
             for name, matches_ in itertools.chain(
-                    function_matches.items(), bb_matches.items(), insn_matches.items()
+                code_capabilities.function_matches.items(),
+                code_capabilities.basic_block_matches.items(),
+                code_capabilities.instruction_matches.items(),
             ):
-                    # in practice, most matches are derived rules,
-                    # like "check OS version/5bf4c7f39fd4492cbed0f6dc7d596d49"
-                    # but when we log to the human, they really care about "real" rules.
                 if not ruleset.rules[name].is_subscope_rule():
                     match_count += len(matches_)
 
             logger.debug(
                 "analyzed function 0x%x and extracted %d features, %d matches in %0.02fs",
                 f.address,
-                    feature_count,
+                code_capabilities.feature_count,
                 match_count,
                 t1 - t0,
             )
 
-                for rule_name, res in function_matches.items():
+            for rule_name, res in code_capabilities.function_matches.items():
                 all_function_matches[rule_name].extend(res)
-                for rule_name, res in bb_matches.items():
+            for rule_name, res in code_capabilities.basic_block_matches.items():
                 all_bb_matches[rule_name].extend(res)
-                for rule_name, res in insn_matches.items():
+            for rule_name, res in code_capabilities.instruction_matches.items():
                 all_insn_matches[rule_name].extend(res)
 
+            pbar.advance(task)
+
     # collection of features that captures the rule matches within function, BB, and instruction scopes.
     # mapping from feature (matched rule) to set of addresses at which it matched.
     function_and_lower_features: FeatureSet = collections.defaultdict(set)
@@ -223,8 +229,8 @@ def find_static_capabilities(
         rule = ruleset[rule_name]
         capa.engine.index_rule_matches(function_and_lower_features, rule, locations)
 
-    all_file_matches, feature_count = find_file_capabilities(ruleset, extractor, function_and_lower_features)
+    all_file_capabilities = find_file_capabilities(ruleset, extractor, function_and_lower_features)
-    feature_counts.file = feature_count
+    feature_counts.file = all_file_capabilities.feature_count
 
     matches: MatchResults = dict(
         itertools.chain(
@@ -234,13 +240,8 @@ def find_static_capabilities(
             all_insn_matches.items(),
             all_bb_matches.items(),
             all_function_matches.items(),
-            all_file_matches.items(),
+            all_file_capabilities.matches.items(),
         )
     )
 
-    meta = {
+    return Capabilities(matches, feature_counts, library_functions)
-        "feature_counts": feature_counts,
-        "library_functions": library_functions,
-    }
-
-    return matches, meta

capa/engine.py

@@ -1,14 +1,21 @@
-# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
+# Copyright 2020 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# You may obtain a copy of the License at
-# Unless required by applicable law or agreed to in writing, software distributed under the License
+#
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#     http://www.apache.org/licenses/LICENSE-2.0
-# See the License for the specific language governing permissions and limitations under the License.
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 
 import copy
 import collections
-from typing import TYPE_CHECKING, Set, Dict, List, Tuple, Union, Mapping, Iterable, Iterator
+from typing import TYPE_CHECKING, Union, Mapping, Iterable, Iterator
 
 import capa.perf
 import capa.features.common
@@ -27,7 +34,7 @@ if TYPE_CHECKING:
 # to collect the locations of a feature, do: `features[Number(0x10)]`
 #
 # aliased here so that the type can be documented and xref'd.
-FeatureSet = Dict[Feature, Set[Address]]
+FeatureSet = dict[Feature, set[Address]]
 
 
 class Statement:
@@ -94,7 +101,7 @@ class And(Statement):
     match if all of the children evaluate to True.
 
     the order of evaluation is dictated by the property
-    `And.children` (type: List[Statement|Feature]).
+    `And.children` (type: list[Statement|Feature]).
     a query optimizer may safely manipulate the order of these children.
     """
 
@@ -127,7 +134,7 @@ class Or(Statement):
     match if any of the children evaluate to True.
 
     the order of evaluation is dictated by the property
-    `Or.children` (type: List[Statement|Feature]).
+    `Or.children` (type: list[Statement|Feature]).
     a query optimizer may safely manipulate the order of these children.
     """
 
@@ -176,7 +183,7 @@ class Some(Statement):
     match if at least N of the children evaluate to True.
 
     the order of evaluation is dictated by the property
-    `Some.children` (type: List[Statement|Feature]).
+    `Some.children` (type: list[Statement|Feature]).
     a query optimizer may safely manipulate the order of these children.
     """
 
@@ -267,7 +274,7 @@ class Subscope(Statement):
 #         inspect(match_details)
 #
 # aliased here so that the type can be documented and xref'd.
-MatchResults = Mapping[str, List[Tuple[Address, Result]]]
+MatchResults = Mapping[str, list[tuple[Address, Result]]]
 
 
 def get_rule_namespaces(rule: "capa.rules.Rule") -> Iterator[str]:
@@ -292,7 +299,7 @@ def index_rule_matches(features: FeatureSet, rule: "capa.rules.Rule", locations:
         features[capa.features.common.MatchedRule(namespace)].update(locations)
 
 
-def match(rules: List["capa.rules.Rule"], features: FeatureSet, addr: Address) -> Tuple[FeatureSet, MatchResults]:
+def match(rules: list["capa.rules.Rule"], features: FeatureSet, addr: Address) -> tuple[FeatureSet, MatchResults]:
     """
     match the given rules against the given features,
     returning an updated set of features and the matches.

capa/exceptions.py

@@ -1,10 +1,18 @@
-# Copyright (C) 2022 Mandiant, Inc. All Rights Reserved.
+# Copyright 2022 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# You may obtain a copy of the License at
-# Unless required by applicable law or agreed to in writing, software distributed under the License
+#
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#     http://www.apache.org/licenses/LICENSE-2.0
-# See the License for the specific language governing permissions and limitations under the License.
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
 class UnsupportedRuntimeError(RuntimeError):
     pass
 
@@ -23,3 +31,15 @@ class UnsupportedOSError(ValueError):
 
 class EmptyReportError(ValueError):
     pass
+
+
+class InvalidArgument(ValueError):
+    pass
+
+
+class NonExistantFunctionError(ValueError):
+    pass
+
+
+class NonExistantProcessError(ValueError):
+    pass

capa/features/address.py

@@ -1,10 +1,17 @@
-# Copyright (C) 2022 Mandiant, Inc. All Rights Reserved.
+# Copyright 2022 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# You may obtain a copy of the License at
-# Unless required by applicable law or agreed to in writing, software distributed under the License
+#
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#     http://www.apache.org/licenses/LICENSE-2.0
-# See the License for the specific language governing permissions and limitations under the License.
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 import abc
 
 
@@ -107,8 +114,7 @@ class DynamicCallAddress(Address):
         return hash((self.thread, self.id))
 
     def __eq__(self, other):
-        assert isinstance(other, DynamicCallAddress)
+        return isinstance(other, DynamicCallAddress) and (self.thread, self.id) == (other.thread, other.id)
-        return (self.thread, self.id) == (other.thread, other.id)
 
     def __lt__(self, other):
         assert isinstance(other, DynamicCallAddress)

capa/features/basicblock.py

@@ -1,10 +1,17 @@
-# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
+# Copyright 2020 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# You may obtain a copy of the License at
-# Unless required by applicable law or agreed to in writing, software distributed under the License
+#
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#     http://www.apache.org/licenses/LICENSE-2.0
-# See the License for the specific language governing permissions and limitations under the License.
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 
 from capa.features.common import Feature
 

capa/features/com/__init__.py

@@ -1,12 +1,18 @@
-# Copyright (C) 2024 Mandiant, Inc. All Rights Reserved.
+# Copyright 2024 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# You may obtain a copy of the License at
-# Unless required by applicable law or agreed to in writing, software distributed under the License
+#
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#     http://www.apache.org/licenses/LICENSE-2.0
-# See the License for the specific language governing permissions and limitations under the License.
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 from enum import Enum
-from typing import Dict, List
 
 from capa.helpers import assert_never
 
@@ -22,7 +28,7 @@ COM_PREFIXES = {
 }
 
 
-def load_com_database(com_type: ComType) -> Dict[str, List[str]]:
+def load_com_database(com_type: ComType) -> dict[str, list[str]]:
     # lazy load these python files since they are so large.
     # that is, don't load them unless a COM feature is being handled.
     import capa.features.com.classes

capa/features/com/classes.py

@@ -1,13 +1,19 @@
-# Copyright (C) 2024 Mandiant, Inc. All Rights Reserved.
+# Copyright 2024 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# You may obtain a copy of the License at
-# Unless required by applicable law or agreed to in writing, software distributed under the License
+#
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#     http://www.apache.org/licenses/LICENSE-2.0
-# See the License for the specific language governing permissions and limitations under the License.
+#
-from typing import Dict, List
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
 
-COM_CLASSES: Dict[str, List[str]] = {
+
+COM_CLASSES: dict[str, list[str]] = {
     "ClusAppWiz": ["24F97150-6689-11D1-9AA7-00C04FB93A80"],
     "ClusCfgAddNodesWizard": ["BB8D141E-C00A-469F-BC5C-ECD814F0BD74"],
     "ClusCfgCreateClusterWizard": ["B929818E-F5B0-44DC-8A00-1B5F5F5AA1F0"],

capa/features/com/interfaces.py

@@ -1,13 +1,19 @@
-# Copyright (C) 2024 Mandiant, Inc. All Rights Reserved.
+# Copyright 2024 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# You may obtain a copy of the License at
-# Unless required by applicable law or agreed to in writing, software distributed under the License
+#
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#     http://www.apache.org/licenses/LICENSE-2.0
-# See the License for the specific language governing permissions and limitations under the License.
+#
-from typing import Dict, List
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
 
-COM_INTERFACES: Dict[str, List[str]] = {
+
+COM_INTERFACES: dict[str, list[str]] = {
     "IClusterApplicationWizard": ["24F97151-6689-11D1-9AA7-00C04FB93A80"],
     "IWEExtendWizard97": ["97DEDE68-FC6B-11CF-B5F5-00A0C90AB505"],
     "IWCWizard97Callback": ["97DEDE67-FC6B-11CF-B5F5-00A0C90AB505"],
@@ -16334,7 +16340,7 @@ COM_INTERFACES: Dict[str, List[str]] = {
     "IRcsServiceDescription": ["416437de-e78b-44c9-990f-7ede1f2a0c91"],
     "IRcsServiceKindSupportedChangedEventArgs": ["f47ea244-e783-4866-b3a7-4e5ccf023070"],
     "IRcsServiceStatusChangedArgs": ["661ae45a-412a-460d-bdd4-dd8ea3c15583"],
-    "IRcsServiceTuple": ["ce17a39b-2e8b-41af-b5a9-5cb072cc373c"],
+    "IRcsServicetuple": ["ce17a39b-2e8b-41af-b5a9-5cb072cc373c"],
     "IRcsSubscriptionReceivedArgs": ["04eaf06d-42bc-46cc-a637-eeb3a8723fe4"],
     "IRcsTransport": ["fea34759-f37c-4319-8546-ec84d21d30ff"],
     "IRcsTransportConfiguration": ["1fccb102-2472-4bb9-9988-c1211c83e8a9"],

capa/features/common.py

@@ -1,18 +1,24 @@
-# Copyright (C) 2021 Mandiant, Inc. All Rights Reserved.
+# Copyright 2021 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# You may obtain a copy of the License at
-# Unless required by applicable law or agreed to in writing, software distributed under the License
+#
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#     http://www.apache.org/licenses/LICENSE-2.0
-# See the License for the specific language governing permissions and limitations under the License.
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 
 import re
 import abc
 import codecs
-import typing
 import logging
 import collections
-from typing import TYPE_CHECKING, Set, Dict, List, Union, Optional
+from typing import TYPE_CHECKING, Union, Optional
 
 if TYPE_CHECKING:
     # circular import, otherwise
@@ -79,14 +85,14 @@ class Result:
         self,
         success: bool,
         statement: Union["capa.engine.Statement", "Feature"],
-        children: List["Result"],
+        children: list["Result"],
-        locations: Optional[Set[Address]] = None,
+        locations: Optional[set[Address]] = None,
     ):
         super().__init__()
         self.success = success
         self.statement = statement
         self.children = children
-        self.locations = locations if locations is not None else set()
+        self.locations = frozenset(locations) if locations is not None else frozenset()
 
     def __eq__(self, other):
         if isinstance(other, bool):
@@ -99,6 +105,25 @@ class Result:
     def __nonzero__(self):
         return self.success
 
+    def __str__(self):
+        # as this object isn't user facing, this formatting is just to help with debugging
+
+        lines: list[str] = []
+
+        def rec(m: "Result", indent: int):
+            if isinstance(m.statement, capa.engine.Statement):
+                line = ("  " * indent) + str(m.statement.name) + " " + str(m.success)
+            else:
+                line = ("  " * indent) + str(m.statement) + " " + str(m.success) + " " + str(m.locations)
+
+            lines.append(line)
+
+            for child in m.children:
+                rec(child, indent + 1)
+
+        rec(self, 0)
+        return "\n".join(lines)
+
 
 class Feature(abc.ABC):  # noqa: B024
     # this is an abstract class, since we don't want anyone to instantiate it directly,
@@ -169,7 +194,11 @@ class Feature(abc.ABC):  # noqa: B024
     def evaluate(self, features: "capa.engine.FeatureSet", short_circuit=True) -> Result:
         capa.perf.counters["evaluate.feature"] += 1
         capa.perf.counters["evaluate.feature." + self.name] += 1
-        return Result(self in features, self, [], locations=features.get(self, set()))
+        success = self in features
+        if success:
+            return Result(True, self, [], locations=features[self])
+        else:
+            return Result(False, self, [], locations=None)
 
 
 class MatchedRule(Feature):
@@ -213,7 +242,7 @@ class Substring(String):
 
         # mapping from string value to list of locations.
         # will unique the locations later on.
-        matches: typing.DefaultDict[str, Set[Address]] = collections.defaultdict(set)
+        matches: collections.defaultdict[str, set[Address]] = collections.defaultdict(set)
 
         assert isinstance(self.value, str)
         for feature, locations in features.items():
@@ -261,7 +290,7 @@ class _MatchedSubstring(Substring):
     note: this type should only ever be constructed by `Substring.evaluate()`. it is not part of the public API.
     """
 
-    def __init__(self, substring: Substring, matches: Dict[str, Set[Address]]):
+    def __init__(self, substring: Substring, matches: dict[str, set[Address]]):
         """
         args:
           substring: the substring feature that matches.
@@ -305,7 +334,7 @@ class Regex(String):
 
         # mapping from string value to list of locations.
         # will unique the locations later on.
-        matches: typing.DefaultDict[str, Set[Address]] = collections.defaultdict(set)
+        matches: collections.defaultdict[str, set[Address]] = collections.defaultdict(set)
 
         for feature, locations in features.items():
             if not isinstance(feature, (String,)):
@@ -353,7 +382,7 @@ class _MatchedRegex(Regex):
     note: this type should only ever be constructed by `Regex.evaluate()`. it is not part of the public API.
     """
 
-    def __init__(self, regex: Regex, matches: Dict[str, Set[Address]]):
+    def __init__(self, regex: Regex, matches: dict[str, set[Address]]):
         """
         args:
           regex: the regex feature that matches.
@@ -424,10 +453,11 @@ class Arch(Feature):
 OS_WINDOWS = "windows"
 OS_LINUX = "linux"
 OS_MACOS = "macos"
+OS_ANDROID = "android"
 # dotnet
 OS_ANY = "any"
 VALID_OS = {os.value for os in capa.features.extractors.elf.OS}
-VALID_OS.update({OS_WINDOWS, OS_LINUX, OS_MACOS, OS_ANY})
+VALID_OS.update({OS_WINDOWS, OS_LINUX, OS_MACOS, OS_ANY, OS_ANDROID})
 # internal only, not to be used in rules
 OS_AUTO = "auto"
 
@@ -461,8 +491,12 @@ FORMAT_AUTO = "auto"
 FORMAT_SC32 = "sc32"
 FORMAT_SC64 = "sc64"
 FORMAT_CAPE = "cape"
+FORMAT_DRAKVUF = "drakvuf"
+FORMAT_VMRAY = "vmray"
+FORMAT_BINEXPORT2 = "binexport2"
 FORMAT_FREEZE = "freeze"
 FORMAT_RESULT = "result"
+FORMAT_BINJA_DB = "binja_database"
 STATIC_FORMATS = {
     FORMAT_SC32,
     FORMAT_SC64,
@@ -471,9 +505,13 @@ STATIC_FORMATS = {
     FORMAT_DOTNET,
     FORMAT_FREEZE,
     FORMAT_RESULT,
+    FORMAT_BINEXPORT2,
+    FORMAT_BINJA_DB,
 }
 DYNAMIC_FORMATS = {
     FORMAT_CAPE,
+    FORMAT_DRAKVUF,
+    FORMAT_VMRAY,
     FORMAT_FREEZE,
     FORMAT_RESULT,
 }

capa/features/extractors/base_extractor.py

@@ -1,21 +1,26 @@
-# Copyright (C) 2021 Mandiant, Inc. All Rights Reserved.
+# Copyright 2021 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# You may obtain a copy of the License at
-# Unless required by applicable law or agreed to in writing, software distributed under the License
+#
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#     http://www.apache.org/licenses/LICENSE-2.0
-# See the License for the specific language governing permissions and limitations under the License.
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 
 import abc
 import hashlib
 import dataclasses
-from typing import Any, Dict, Tuple, Union, Iterator
+from copy import copy
+from types import MethodType
+from typing import Any, Union, Iterator, TypeAlias
 from dataclasses import dataclass
 
-# TODO(williballenthin): use typing.TypeAlias directly when Python 3.9 is deprecated
-# https://github.com/mandiant/capa/issues/1699
-from typing_extensions import TypeAlias
-
 import capa.features.address
 from capa.features.common import Feature
 from capa.features.address import Address, ThreadAddress, ProcessAddress, DynamicCallAddress, AbsoluteVirtualAddress
@@ -57,7 +62,7 @@ class FunctionHandle:
 
     address: Address
     inner: Any
-    ctx: Dict[str, Any] = dataclasses.field(default_factory=dict)
+    ctx: dict[str, Any] = dataclasses.field(default_factory=dict)
 
 
 @dataclass
@@ -133,7 +138,7 @@ class StaticFeatureExtractor:
         return self._sample_hashes
 
     @abc.abstractmethod
-    def extract_global_features(self) -> Iterator[Tuple[Feature, Address]]:
+    def extract_global_features(self) -> Iterator[tuple[Feature, Address]]:
         """
         extract features found at every scope ("global").
 
@@ -144,12 +149,12 @@ class StaticFeatureExtractor:
                 print('0x%x: %s', va, feature)
 
         yields:
-          Tuple[Feature, Address]: feature and its location
+          tuple[Feature, Address]: feature and its location
         """
         raise NotImplementedError()
 
     @abc.abstractmethod
-    def extract_file_features(self) -> Iterator[Tuple[Feature, Address]]:
+    def extract_file_features(self) -> Iterator[tuple[Feature, Address]]:
         """
         extract file-scope features.
 
@@ -160,7 +165,7 @@ class StaticFeatureExtractor:
                 print('0x%x: %s', va, feature)
 
         yields:
-          Tuple[Feature, Address]: feature and its location
+          tuple[Feature, Address]: feature and its location
         """
         raise NotImplementedError()
 
@@ -209,7 +214,7 @@ class StaticFeatureExtractor:
         raise KeyError(addr)
 
     @abc.abstractmethod
-    def extract_function_features(self, f: FunctionHandle) -> Iterator[Tuple[Feature, Address]]:
+    def extract_function_features(self, f: FunctionHandle) -> Iterator[tuple[Feature, Address]]:
         """
         extract function-scope features.
         the arguments are opaque values previously provided by `.get_functions()`, etc.
@@ -225,7 +230,7 @@ class StaticFeatureExtractor:
           f [FunctionHandle]: an opaque value previously fetched from `.get_functions()`.
 
         yields:
-          Tuple[Feature, Address]: feature and its location
+          tuple[Feature, Address]: feature and its location
         """
         raise NotImplementedError()
 
@@ -238,7 +243,7 @@ class StaticFeatureExtractor:
         raise NotImplementedError()
 
     @abc.abstractmethod
-    def extract_basic_block_features(self, f: FunctionHandle, bb: BBHandle) -> Iterator[Tuple[Feature, Address]]:
+    def extract_basic_block_features(self, f: FunctionHandle, bb: BBHandle) -> Iterator[tuple[Feature, Address]]:
         """
         extract basic block-scope features.
         the arguments are opaque values previously provided by `.get_functions()`, etc.
@@ -256,7 +261,7 @@ class StaticFeatureExtractor:
           bb [BBHandle]: an opaque value previously fetched from `.get_basic_blocks()`.
 
         yields:
-          Tuple[Feature, Address]: feature and its location
+          tuple[Feature, Address]: feature and its location
         """
         raise NotImplementedError()
 
@@ -271,7 +276,7 @@ class StaticFeatureExtractor:
     @abc.abstractmethod
     def extract_insn_features(
         self, f: FunctionHandle, bb: BBHandle, insn: InsnHandle
-    ) -> Iterator[Tuple[Feature, Address]]:
+    ) -> Iterator[tuple[Feature, Address]]:
         """
         extract instruction-scope features.
         the arguments are opaque values previously provided by `.get_functions()`, etc.
@@ -291,11 +296,27 @@ class StaticFeatureExtractor:
           insn [InsnHandle]: an opaque value previously fetched from `.get_instructions()`.
 
         yields:
-          Tuple[Feature, Address]: feature and its location
+          tuple[Feature, Address]: feature and its location
         """
         raise NotImplementedError()
 
 
+def FunctionFilter(extractor: StaticFeatureExtractor, functions: set) -> StaticFeatureExtractor:
+    original_get_functions = extractor.get_functions
+
+    def filtered_get_functions(self):
+        yield from (f for f in original_get_functions() if f.address in functions)
+
+    # we make a copy of the original extractor object and then update its get_functions() method with the decorated filter one.
+    # this is in order to preserve the original extractor object's get_functions() method, in case it is used elsewhere in the code.
+    # an example where this is important is in our testfiles where we may use the same extractor object with different tests,
+    # with some of these tests needing to install a functions filter on the extractor object.
+    new_extractor = copy(extractor)
+    new_extractor.get_functions = MethodType(filtered_get_functions, extractor)  # type: ignore
+
+    return new_extractor
+
+
 @dataclass
 class ProcessHandle:
     """
@@ -369,7 +390,7 @@ class DynamicFeatureExtractor:
         return self._sample_hashes
 
     @abc.abstractmethod
-    def extract_global_features(self) -> Iterator[Tuple[Feature, Address]]:
+    def extract_global_features(self) -> Iterator[tuple[Feature, Address]]:
         """
         extract features found at every scope ("global").
 
@@ -380,12 +401,12 @@ class DynamicFeatureExtractor:
                 print(addr, feature)
 
         yields:
-          Tuple[Feature, Address]: feature and its location
+          tuple[Feature, Address]: feature and its location
         """
         raise NotImplementedError()
 
     @abc.abstractmethod
-    def extract_file_features(self) -> Iterator[Tuple[Feature, Address]]:
+    def extract_file_features(self) -> Iterator[tuple[Feature, Address]]:
         """
         extract file-scope features.
 
@@ -396,7 +417,7 @@ class DynamicFeatureExtractor:
                 print(addr, feature)
 
         yields:
-          Tuple[Feature, Address]: feature and its location
+          tuple[Feature, Address]: feature and its location
         """
         raise NotImplementedError()
 
@@ -408,7 +429,7 @@ class DynamicFeatureExtractor:
         raise NotImplementedError()
 
     @abc.abstractmethod
-    def extract_process_features(self, ph: ProcessHandle) -> Iterator[Tuple[Feature, Address]]:
+    def extract_process_features(self, ph: ProcessHandle) -> Iterator[tuple[Feature, Address]]:
         """
         Yields all the features of a process. These include:
         - file features of the process' image
@@ -431,7 +452,7 @@ class DynamicFeatureExtractor:
         raise NotImplementedError()
 
     @abc.abstractmethod
-    def extract_thread_features(self, ph: ProcessHandle, th: ThreadHandle) -> Iterator[Tuple[Feature, Address]]:
+    def extract_thread_features(self, ph: ProcessHandle, th: ThreadHandle) -> Iterator[tuple[Feature, Address]]:
         """
         Yields all the features of a thread. These include:
         - sequenced api traces
@@ -448,7 +469,7 @@ class DynamicFeatureExtractor:
     @abc.abstractmethod
     def extract_call_features(
         self, ph: ProcessHandle, th: ThreadHandle, ch: CallHandle
-    ) -> Iterator[Tuple[Feature, Address]]:
+    ) -> Iterator[tuple[Feature, Address]]:
         """
         Yields all features of a call. These include:
         - api name
@@ -467,4 +488,32 @@ class DynamicFeatureExtractor:
         raise NotImplementedError()
 
 
+def ProcessFilter(extractor: DynamicFeatureExtractor, pids: set[int]) -> DynamicFeatureExtractor:
+    original_get_processes = extractor.get_processes
+
+    def filtered_get_processes(self):
+        yield from (f for f in original_get_processes() if f.address.pid in pids)
+
+    # we make a copy of the original extractor object and then update its get_processes() method with the decorated filter one.
+    # this is in order to preserve the original extractor object's get_processes() method, in case it is used elsewhere in the code.
+    # an example where this is important is in our testfiles where we may use the same extractor object with different tests,
+    # with some of these tests needing to install a processes filter on the extractor object.
+    new_extractor = copy(extractor)
+    new_extractor.get_processes = MethodType(filtered_get_processes, extractor)  # type: ignore
+
+    return new_extractor
+
+
+def ThreadFilter(extractor: DynamicFeatureExtractor, threads: set[Address]) -> DynamicFeatureExtractor:
+    original_get_threads = extractor.get_threads
+
+    def filtered_get_threads(self, ph: ProcessHandle):
+        yield from (t for t in original_get_threads(ph) if t.address in threads)
+
+    new_extractor = copy(extractor)
+    new_extractor.get_threads = MethodType(filtered_get_threads, extractor)  # type: ignore
+
+    return new_extractor
+
+
 FeatureExtractor: TypeAlias = Union[StaticFeatureExtractor, DynamicFeatureExtractor]

capa/features/extractors/binexport2/__init__.py

@@ -0,0 +1,425 @@
+# Copyright 2023 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Proto files generated via protobuf v24.4:
+
+    protoc --python_out=. --mypy_out=. binexport2.proto
+
+from BinExport2 at 6916731d5f6693c4a4f0a052501fd3bd92cfd08b
+https://github.com/google/binexport/blob/6916731/binexport2.proto
+"""
+import io
+import hashlib
+import logging
+import contextlib
+from typing import Iterator
+from pathlib import Path
+from collections import defaultdict
+from dataclasses import dataclass
+
+from pefile import PE
+from elftools.elf.elffile import ELFFile
+
+import capa.features.common
+import capa.features.extractors.common
+import capa.features.extractors.binexport2.helpers
+from capa.features.extractors.binexport2.binexport2_pb2 import BinExport2
+
+logger = logging.getLogger(__name__)
+
+
+def get_binexport2(sample: Path) -> BinExport2:
+    be2: BinExport2 = BinExport2()
+    be2.ParseFromString(sample.read_bytes())
+    return be2
+
+
+def compute_common_prefix_length(m: str, n: str) -> int:
+    # ensure #m < #n
+    if len(n) < len(m):
+        m, n = n, m
+
+    for i, c in enumerate(m):
+        if n[i] != c:
+            return i
+
+    return len(m)
+
+
+def get_sample_from_binexport2(input_file: Path, be2: BinExport2, search_paths: list[Path]) -> Path:
+    """attempt to find the sample file, given a BinExport2 file.
+
+    searches in the same directory as the BinExport2 file, and then in search_paths.
+    """
+
+    def filename_similarity_key(p: Path) -> tuple[int, str]:
+        # note closure over input_file.
+        # sort first by length of common prefix, then by name (for stability)
+        return (compute_common_prefix_length(p.name, input_file.name), p.name)
+
+    wanted_sha256: str = be2.meta_information.executable_id.lower()
+
+    input_directory: Path = input_file.parent
+    siblings: list[Path] = [p for p in input_directory.iterdir() if p.is_file()]
+    siblings.sort(key=filename_similarity_key, reverse=True)
+    for sibling in siblings:
+        # e.g. with open IDA files in the same directory on Windows
+        with contextlib.suppress(PermissionError):
+            if hashlib.sha256(sibling.read_bytes()).hexdigest().lower() == wanted_sha256:
+                return sibling
+
+    for search_path in search_paths:
+        candidates: list[Path] = [p for p in search_path.iterdir() if p.is_file()]
+        candidates.sort(key=filename_similarity_key, reverse=True)
+        for candidate in candidates:
+            with contextlib.suppress(PermissionError):
+                if hashlib.sha256(candidate.read_bytes()).hexdigest().lower() == wanted_sha256:
+                    return candidate
+
+    raise ValueError("cannot find sample, you may specify the path using the CAPA_SAMPLES_DIR environment variable")
+
+
+class BinExport2Index:
+    def __init__(self, be2: BinExport2):
+        self.be2: BinExport2 = be2
+
+        self.callers_by_vertex_index: dict[int, list[int]] = defaultdict(list)
+        self.callees_by_vertex_index: dict[int, list[int]] = defaultdict(list)
+
+        # note: flow graph != call graph (vertex)
+        self.flow_graph_index_by_address: dict[int, int] = {}
+        self.flow_graph_address_by_index: dict[int, int] = {}
+
+        # edges that come from the given basic block
+        self.source_edges_by_basic_block_index: dict[int, list[BinExport2.FlowGraph.Edge]] = defaultdict(list)
+        # edges that end up at the given basic block
+        self.target_edges_by_basic_block_index: dict[int, list[BinExport2.FlowGraph.Edge]] = defaultdict(list)
+
+        self.vertex_index_by_address: dict[int, int] = {}
+
+        self.data_reference_index_by_source_instruction_index: dict[int, list[int]] = defaultdict(list)
+        self.data_reference_index_by_target_address: dict[int, list[int]] = defaultdict(list)
+        self.string_reference_index_by_source_instruction_index: dict[int, list[int]] = defaultdict(list)
+
+        self.insn_address_by_index: dict[int, int] = {}
+        self.insn_index_by_address: dict[int, int] = {}
+        self.insn_by_address: dict[int, BinExport2.Instruction] = {}
+
+        # must index instructions first
+        self._index_insn_addresses()
+        self._index_vertex_edges()
+        self._index_flow_graph_nodes()
+        self._index_flow_graph_edges()
+        self._index_call_graph_vertices()
+        self._index_data_references()
+        self._index_string_references()
+
+    def get_insn_address(self, insn_index: int) -> int:
+        assert insn_index in self.insn_address_by_index, f"insn must be indexed, missing {insn_index}"
+        return self.insn_address_by_index[insn_index]
+
+    def get_basic_block_address(self, basic_block_index: int) -> int:
+        basic_block: BinExport2.BasicBlock = self.be2.basic_block[basic_block_index]
+        first_instruction_index: int = next(self.instruction_indices(basic_block))
+        return self.get_insn_address(first_instruction_index)
+
+    def _index_vertex_edges(self):
+        for edge in self.be2.call_graph.edge:
+            if not edge.source_vertex_index:
+                continue
+            if not edge.target_vertex_index:
+                continue
+
+            self.callers_by_vertex_index[edge.target_vertex_index].append(edge.source_vertex_index)
+            self.callees_by_vertex_index[edge.source_vertex_index].append(edge.target_vertex_index)
+
+    def _index_flow_graph_nodes(self):
+        for flow_graph_index, flow_graph in enumerate(self.be2.flow_graph):
+            function_address: int = self.get_basic_block_address(flow_graph.entry_basic_block_index)
+            self.flow_graph_index_by_address[function_address] = flow_graph_index
+            self.flow_graph_address_by_index[flow_graph_index] = function_address
+
+    def _index_flow_graph_edges(self):
+        for flow_graph in self.be2.flow_graph:
+            for edge in flow_graph.edge:
+                if not edge.HasField("source_basic_block_index") or not edge.HasField("target_basic_block_index"):
+                    continue
+
+                self.source_edges_by_basic_block_index[edge.source_basic_block_index].append(edge)
+                self.target_edges_by_basic_block_index[edge.target_basic_block_index].append(edge)
+
+    def _index_call_graph_vertices(self):
+        for vertex_index, vertex in enumerate(self.be2.call_graph.vertex):
+            if not vertex.HasField("address"):
+                continue
+
+            vertex_address: int = vertex.address
+            self.vertex_index_by_address[vertex_address] = vertex_index
+
+    def _index_data_references(self):
+        for data_reference_index, data_reference in enumerate(self.be2.data_reference):
+            self.data_reference_index_by_source_instruction_index[data_reference.instruction_index].append(
+                data_reference_index
+            )
+            self.data_reference_index_by_target_address[data_reference.address].append(data_reference_index)
+
+    def _index_string_references(self):
+        for string_reference_index, string_reference in enumerate(self.be2.string_reference):
+            self.string_reference_index_by_source_instruction_index[string_reference.instruction_index].append(
+                string_reference_index
+            )
+
+    def _index_insn_addresses(self):
+        # see https://github.com/google/binexport/blob/39f6445c232bb5caf5c4a2a996de91dfa20c48e8/binexport.cc#L45
+        if len(self.be2.instruction) == 0:
+            return
+
+        assert self.be2.instruction[0].HasField("address"), "first insn must have explicit address"
+
+        addr: int = 0
+        next_addr: int = 0
+        for idx, insn in enumerate(self.be2.instruction):
+            if insn.HasField("address"):
+                addr = insn.address
+                next_addr = addr + len(insn.raw_bytes)
+            else:
+                addr = next_addr
+                next_addr += len(insn.raw_bytes)
+            self.insn_address_by_index[idx] = addr
+            self.insn_index_by_address[addr] = idx
+            self.insn_by_address[addr] = insn
+
+    @staticmethod
+    def instruction_indices(basic_block: BinExport2.BasicBlock) -> Iterator[int]:
+        """
+        For a given basic block, enumerate the instruction indices.
+        """
+        for index_range in basic_block.instruction_index:
+            if not index_range.HasField("end_index"):
+                yield index_range.begin_index
+                continue
+            else:
+                yield from range(index_range.begin_index, index_range.end_index)
+
+    def basic_block_instructions(
+        self, basic_block: BinExport2.BasicBlock
+    ) -> Iterator[tuple[int, BinExport2.Instruction, int]]:
+        """
+        For a given basic block, enumerate the instruction indices,
+        the instruction instances, and their addresses.
+        """
+        for instruction_index in self.instruction_indices(basic_block):
+            instruction: BinExport2.Instruction = self.be2.instruction[instruction_index]
+            instruction_address: int = self.get_insn_address(instruction_index)
+
+            yield instruction_index, instruction, instruction_address
+
+    def get_function_name_by_vertex(self, vertex_index: int) -> str:
+        vertex: BinExport2.CallGraph.Vertex = self.be2.call_graph.vertex[vertex_index]
+        name: str = f"sub_{vertex.address:x}"
+        if vertex.HasField("mangled_name"):
+            name = vertex.mangled_name
+
+        if vertex.HasField("demangled_name"):
+            name = vertex.demangled_name
+
+        if vertex.HasField("library_index"):
+            library: BinExport2.Library = self.be2.library[vertex.library_index]
+            if library.HasField("name"):
+                name = f"{library.name}!{name}"
+
+        return name
+
+    def get_function_name_by_address(self, address: int) -> str:
+        if address not in self.vertex_index_by_address:
+            return ""
+
+        vertex_index: int = self.vertex_index_by_address[address]
+        return self.get_function_name_by_vertex(vertex_index)
+
+    def get_instruction_by_address(self, address: int) -> BinExport2.Instruction:
+        assert address in self.insn_by_address, f"address must be indexed, missing {address:x}"
+        return self.insn_by_address[address]
+
+
+class BinExport2Analysis:
+    def __init__(self, be2: BinExport2, idx: BinExport2Index, buf: bytes):
+        self.be2: BinExport2 = be2
+        self.idx: BinExport2Index = idx
+        self.buf: bytes = buf
+        self.base_address: int = 0
+        self.thunks: dict[int, int] = {}
+
+        self._find_base_address()
+        self._compute_thunks()
+
+    def _find_base_address(self):
+        sections_with_perms: Iterator[BinExport2.Section] = filter(
+            lambda s: s.flag_r or s.flag_w or s.flag_x, self.be2.section
+        )
+        # assume the lowest address is the base address.
+        # this works as long as BinExport doesn't record other
+        # libraries mapped into memory.
+        self.base_address = min(s.address for s in sections_with_perms)
+
+        logger.debug("found base address: %x", self.base_address)
+
+    def _compute_thunks(self):
+        for addr, idx in self.idx.vertex_index_by_address.items():
+            vertex: BinExport2.CallGraph.Vertex = self.be2.call_graph.vertex[idx]
+            if not capa.features.extractors.binexport2.helpers.is_vertex_type(
+                vertex, BinExport2.CallGraph.Vertex.Type.THUNK
+            ):
+                continue
+
+            curr_idx: int = idx
+            for _ in range(capa.features.common.THUNK_CHAIN_DEPTH_DELTA):
+                thunk_callees: list[int] = self.idx.callees_by_vertex_index[curr_idx]
+                # If this doesn't hold, then it doesn't seem like this is a thunk,
+                # because either, len is:
+                #    0 and the thunk doesn't point to anything or is indirect, like `call eax`, or
+                #   >1 and the thunk may end up at many functions.
+                # In any case, this doesn't appear to be the sort of thunk we're looking for.
+                if len(thunk_callees) != 1:
+                    break
+
+                thunked_idx: int = thunk_callees[0]
+                thunked_vertex: BinExport2.CallGraph.Vertex = self.be2.call_graph.vertex[thunked_idx]
+
+                if not capa.features.extractors.binexport2.helpers.is_vertex_type(
+                    thunked_vertex, BinExport2.CallGraph.Vertex.Type.THUNK
+                ):
+                    assert thunked_vertex.HasField("address")
+
+                    self.thunks[addr] = thunked_vertex.address
+                    break
+
+                curr_idx = thunked_idx
+
+
+@dataclass
+class MemoryRegion:
+    # location of the bytes, potentially relative to a base address
+    address: int
+    buf: bytes
+
+    @property
+    def end(self) -> int:
+        return self.address + len(self.buf)
+
+    def contains(self, address: int) -> bool:
+        # note: address must be relative to any base address
+        return self.address <= address < self.end
+
+
+class ReadMemoryError(ValueError): ...
+
+
+class AddressNotMappedError(ReadMemoryError): ...
+
+
+@dataclass
+class AddressSpace:
+    base_address: int
+    memory_regions: tuple[MemoryRegion, ...]
+
+    def read_memory(self, address: int, length: int) -> bytes:
+        rva: int = address - self.base_address
+        for region in self.memory_regions:
+            if region.contains(rva):
+                offset: int = rva - region.address
+                return region.buf[offset : offset + length]
+
+        raise AddressNotMappedError(address)
+
+    @classmethod
+    def from_pe(cls, pe: PE, base_address: int):
+        regions: list[MemoryRegion] = []
+        for section in pe.sections:
+            address: int = section.VirtualAddress
+            size: int = section.Misc_VirtualSize
+            buf: bytes = section.get_data()
+
+            if len(buf) != size:
+                # pad the section with NULLs
+                # assume page alignment is already handled.
+                # might need more hardening here.
+                buf += b"\x00" * (size - len(buf))
+
+            regions.append(MemoryRegion(address, buf))
+
+        return cls(base_address, tuple(regions))
+
+    @classmethod
+    def from_elf(cls, elf: ELFFile, base_address: int):
+        regions: list[MemoryRegion] = []
+
+        # ELF segments are for runtime data,
+        # ELF sections are for link-time data.
+        for segment in elf.iter_segments():
+            # assume p_align is consistent with addresses here.
+            # otherwise, should harden this loader.
+            segment_rva: int = segment.header.p_vaddr
+            segment_size: int = segment.header.p_memsz
+            segment_data: bytes = segment.data()
+
+            if len(segment_data) < segment_size:
+                # pad the section with NULLs
+                # assume page alignment is already handled.
+                # might need more hardening here.
+                segment_data += b"\x00" * (segment_size - len(segment_data))
+
+            regions.append(MemoryRegion(segment_rva, segment_data))
+
+        return cls(base_address, tuple(regions))
+
+    @classmethod
+    def from_buf(cls, buf: bytes, base_address: int):
+        if buf.startswith(capa.features.extractors.common.MATCH_PE):
+            pe: PE = PE(data=buf)
+            return cls.from_pe(pe, base_address)
+        elif buf.startswith(capa.features.extractors.common.MATCH_ELF):
+            elf: ELFFile = ELFFile(io.BytesIO(buf))
+            return cls.from_elf(elf, base_address)
+        else:
+            raise NotImplementedError("file format address space")
+
+
+@dataclass
+class AnalysisContext:
+    sample_bytes: bytes
+    be2: BinExport2
+    idx: BinExport2Index
+    analysis: BinExport2Analysis
+    address_space: AddressSpace
+
+
+@dataclass
+class FunctionContext:
+    ctx: AnalysisContext
+    flow_graph_index: int
+    format: set[str]
+    os: set[str]
+    arch: set[str]
+
+
+@dataclass
+class BasicBlockContext:
+    basic_block_index: int
+
+
+@dataclass
+class InstructionContext:
+    instruction_index: int

capa/features/extractors/binexport2/arch/arm/helpers.py

@@ -0,0 +1,22 @@
+# Copyright 2024 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+from capa.features.extractors.binexport2.binexport2_pb2 import BinExport2
+
+
+def is_stack_register_expression(be2: BinExport2, expression: BinExport2.Expression) -> bool:
+    return bool(
+        expression and expression.type == BinExport2.Expression.REGISTER and expression.symbol.lower().endswith("sp")
+    )

capa/features/extractors/binexport2/arch/arm/insn.py

@@ -0,0 +1,162 @@
+# Copyright 2024 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import logging
+from typing import Iterator, Optional
+
+import capa.features.extractors.binexport2.helpers
+from capa.features.insn import MAX_STRUCTURE_SIZE, Number, Offset, OperandNumber, OperandOffset
+from capa.features.common import Feature, Characteristic
+from capa.features.address import Address
+from capa.features.extractors.binexport2 import FunctionContext, InstructionContext
+from capa.features.extractors.base_extractor import BBHandle, InsnHandle, FunctionHandle
+from capa.features.extractors.binexport2.helpers import (
+    BinExport2InstructionPatternMatcher,
+    mask_immediate,
+    is_address_mapped,
+    get_instruction_mnemonic,
+    get_operand_register_expression,
+    get_operand_immediate_expression,
+)
+from capa.features.extractors.binexport2.binexport2_pb2 import BinExport2
+from capa.features.extractors.binexport2.arch.arm.helpers import is_stack_register_expression
+
+logger = logging.getLogger(__name__)
+
+
+def extract_insn_number_features(
+    fh: FunctionHandle, _bbh: BBHandle, ih: InsnHandle
+) -> Iterator[tuple[Feature, Address]]:
+    fhi: FunctionContext = fh.inner
+    ii: InstructionContext = ih.inner
+
+    be2: BinExport2 = fhi.ctx.be2
+
+    instruction_index: int = ii.instruction_index
+    instruction: BinExport2.Instruction = be2.instruction[instruction_index]
+
+    if len(instruction.operand_index) == 0:
+        # skip things like:
+        #   .text:0040116e leave
+        return
+
+    mnemonic: str = get_instruction_mnemonic(be2, instruction)
+
+    if mnemonic in ("add", "sub"):
+        assert len(instruction.operand_index) == 3
+
+        operand1_expression: Optional[BinExport2.Expression] = get_operand_register_expression(
+            be2, be2.operand[instruction.operand_index[1]]
+        )
+        if operand1_expression and is_stack_register_expression(be2, operand1_expression):
+            # skip things like:
+            # add x0,sp,#0x8
+            return
+
+    for i, operand_index in enumerate(instruction.operand_index):
+        operand: BinExport2.Operand = be2.operand[operand_index]
+
+        immediate_expression: Optional[BinExport2.Expression] = get_operand_immediate_expression(be2, operand)
+        if not immediate_expression:
+            continue
+
+        value: int = mask_immediate(fhi.arch, immediate_expression.immediate)
+        if is_address_mapped(be2, value):
+            continue
+
+        yield Number(value), ih.address
+        yield OperandNumber(i, value), ih.address
+
+        if mnemonic == "add" and i == 2:
+            if 0 < value < MAX_STRUCTURE_SIZE:
+                yield Offset(value), ih.address
+                yield OperandOffset(i, value), ih.address
+
+
+OFFSET_PATTERNS = BinExport2InstructionPatternMatcher.from_str(
+    """
+    ldr|ldrb|ldrh|ldrsb|ldrsh|ldrex|ldrd|str|strb|strh|strex|strd reg, [reg(not-stack),  #int]                                 ; capture #int
+    ldr|ldrb|ldrh|ldrsb|ldrsh|ldrex|ldrd|str|strb|strh|strex|strd reg, [reg(not-stack),  #int]!                                ; capture #int
+    ldr|ldrb|ldrh|ldrsb|ldrsh|ldrex|ldrd|str|strb|strh|strex|strd reg, [reg(not-stack)],        #int                           ; capture #int
+    ldp|ldpd|stp|stpd                                             reg, reg,                     [reg(not-stack), #int]         ; capture #int
+    ldp|ldpd|stp|stpd                                             reg, reg,                     [reg(not-stack), #int]!        ; capture #int
+    ldp|ldpd|stp|stpd                                             reg, reg,                     [reg(not-stack)],       #int   ; capture #int
+    """
+)
+
+
+def extract_insn_offset_features(
+    fh: FunctionHandle, bbh: BBHandle, ih: InsnHandle
+) -> Iterator[tuple[Feature, Address]]:
+    fhi: FunctionContext = fh.inner
+    ii: InstructionContext = ih.inner
+
+    be2: BinExport2 = fhi.ctx.be2
+
+    match = OFFSET_PATTERNS.match_with_be2(be2, ii.instruction_index)
+    if not match:
+        return
+
+    value = match.expression.immediate
+
+    value = mask_immediate(fhi.arch, value)
+    if not is_address_mapped(be2, value):
+        value = capa.features.extractors.binexport2.helpers.twos_complement(fhi.arch, value)
+        yield Offset(value), ih.address
+        yield OperandOffset(match.operand_index, value), ih.address
+
+
+NZXOR_PATTERNS = BinExport2InstructionPatternMatcher.from_str(
+    """
+    eor reg, reg, reg
+    eor reg, reg, #int
+    """
+)
+
+
+def extract_insn_nzxor_characteristic_features(
+    fh: FunctionHandle, bbh: BBHandle, ih: InsnHandle
+) -> Iterator[tuple[Feature, Address]]:
+    fhi: FunctionContext = fh.inner
+    ii: InstructionContext = ih.inner
+    be2: BinExport2 = fhi.ctx.be2
+
+    if NZXOR_PATTERNS.match_with_be2(be2, ii.instruction_index) is None:
+        return
+
+    instruction: BinExport2.Instruction = be2.instruction[ii.instruction_index]
+    # guaranteed to be simple int/reg operands
+    # so we don't have to realize the tree/list.
+    operands: list[BinExport2.Operand] = [be2.operand[operand_index] for operand_index in instruction.operand_index]
+
+    if operands[1] != operands[2]:
+        yield Characteristic("nzxor"), ih.address
+
+
+INDIRECT_CALL_PATTERNS = BinExport2InstructionPatternMatcher.from_str(
+    """
+    blx|bx|blr reg
+    """
+)
+
+
+def extract_function_indirect_call_characteristic_features(
+    fh: FunctionHandle, bbh: BBHandle, ih: InsnHandle
+) -> Iterator[tuple[Feature, Address]]:
+    fhi: FunctionContext = fh.inner
+    ii: InstructionContext = ih.inner
+    be2: BinExport2 = fhi.ctx.be2
+
+    if INDIRECT_CALL_PATTERNS.match_with_be2(be2, ii.instruction_index) is not None:
+        yield Characteristic("indirect call"), ih.address

capa/features/extractors/binexport2/arch/intel/helpers.py

@@ -0,0 +1,142 @@
+# Copyright 2024 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from typing import Optional
+from dataclasses import dataclass
+
+from capa.features.extractors.binexport2.helpers import get_operand_expressions
+from capa.features.extractors.binexport2.binexport2_pb2 import BinExport2
+
+# security cookie checks may perform non-zeroing XORs, these are expected within a certain
+# byte range within the first and returning basic blocks, this helps to reduce FP features
+SECURITY_COOKIE_BYTES_DELTA: int = 0x40
+
+
+@dataclass
+class OperandPhraseInfo:
+    scale: Optional[BinExport2.Expression] = None
+    index: Optional[BinExport2.Expression] = None
+    base: Optional[BinExport2.Expression] = None
+    displacement: Optional[BinExport2.Expression] = None
+
+
+def get_operand_phrase_info(be2: BinExport2, operand: BinExport2.Operand) -> Optional[OperandPhraseInfo]:
+    # assume the following (see https://blog.yossarian.net/2020/06/13/How-x86_64-addresses-memory):
+    #
+    # Scale: A 2-bit constant factor
+    # Index: Any general purpose register
+    # Base: Any general purpose register
+    # Displacement: An integral offset
+
+    expressions: list[BinExport2.Expression] = get_operand_expressions(be2, operand)
+
+    # skip expression up to and including BinExport2.Expression.DEREFERENCE, assume caller
+    # has checked for BinExport2.Expression.DEREFERENCE
+    for i, expression in enumerate(expressions):
+        if expression.type == BinExport2.Expression.DEREFERENCE:
+            expressions = expressions[i + 1 :]
+            break
+
+    expression0: BinExport2.Expression
+    expression1: BinExport2.Expression
+    expression2: BinExport2.Expression
+    expression3: BinExport2.Expression
+    expression4: BinExport2.Expression
+
+    if len(expressions) == 1:
+        expression0 = expressions[0]
+
+        assert (
+            expression0.type == BinExport2.Expression.IMMEDIATE_INT
+            or expression0.type == BinExport2.Expression.REGISTER
+        )
+
+        if expression0.type == BinExport2.Expression.IMMEDIATE_INT:
+            # Displacement
+            return OperandPhraseInfo(displacement=expression0)
+        elif expression0.type == BinExport2.Expression.REGISTER:
+            # Base
+            return OperandPhraseInfo(base=expression0)
+
+    elif len(expressions) == 3:
+        expression0 = expressions[0]
+        expression1 = expressions[1]
+        expression2 = expressions[2]
+
+        assert expression0.type == BinExport2.Expression.REGISTER
+        assert expression1.type == BinExport2.Expression.OPERATOR
+        assert (
+            expression2.type == BinExport2.Expression.IMMEDIATE_INT
+            or expression2.type == BinExport2.Expression.REGISTER
+        )
+
+        if expression2.type == BinExport2.Expression.REGISTER:
+            # Base + Index
+            return OperandPhraseInfo(base=expression0, index=expression2)
+        elif expression2.type == BinExport2.Expression.IMMEDIATE_INT:
+            # Base + Displacement
+            return OperandPhraseInfo(base=expression0, displacement=expression2)
+
+    elif len(expressions) == 5:
+        expression0 = expressions[0]
+        expression1 = expressions[1]
+        expression2 = expressions[2]
+        expression3 = expressions[3]
+        expression4 = expressions[4]
+
+        assert expression0.type == BinExport2.Expression.REGISTER
+        assert expression1.type == BinExport2.Expression.OPERATOR
+        assert (
+            expression2.type == BinExport2.Expression.REGISTER
+            or expression2.type == BinExport2.Expression.IMMEDIATE_INT
+        )
+        assert expression3.type == BinExport2.Expression.OPERATOR
+        assert expression4.type == BinExport2.Expression.IMMEDIATE_INT
+
+        if expression1.symbol == "+" and expression3.symbol == "+":
+            # Base + Index + Displacement
+            return OperandPhraseInfo(base=expression0, index=expression2, displacement=expression4)
+        elif expression1.symbol == "+" and expression3.symbol == "*":
+            # Base + (Index * Scale)
+            return OperandPhraseInfo(base=expression0, index=expression2, scale=expression3)
+        elif expression1.symbol == "*" and expression3.symbol == "+":
+            # (Index * Scale) + Displacement
+            return OperandPhraseInfo(index=expression0, scale=expression2, displacement=expression3)
+        else:
+            raise NotImplementedError(expression1.symbol, expression3.symbol)
+
+    elif len(expressions) == 7:
+        expression0 = expressions[0]
+        expression1 = expressions[1]
+        expression2 = expressions[2]
+        expression3 = expressions[3]
+        expression4 = expressions[4]
+        expression5 = expressions[5]
+        expression6 = expressions[6]
+
+        assert expression0.type == BinExport2.Expression.REGISTER
+        assert expression1.type == BinExport2.Expression.OPERATOR
+        assert expression2.type == BinExport2.Expression.REGISTER
+        assert expression3.type == BinExport2.Expression.OPERATOR
+        assert expression4.type == BinExport2.Expression.IMMEDIATE_INT
+        assert expression5.type == BinExport2.Expression.OPERATOR
+        assert expression6.type == BinExport2.Expression.IMMEDIATE_INT
+
+        # Base + (Index * Scale) + Displacement
+        return OperandPhraseInfo(base=expression0, index=expression2, scale=expression4, displacement=expression6)
+
+    else:
+        raise NotImplementedError(len(expressions))
+
+    return None

capa/features/extractors/binexport2/arch/intel/insn.py

@@ -0,0 +1,255 @@
+# Copyright 2024 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import logging
+from typing import Iterator
+
+import capa.features.extractors.strings
+import capa.features.extractors.binexport2.helpers
+from capa.features.insn import MAX_STRUCTURE_SIZE, Number, Offset, OperandNumber, OperandOffset
+from capa.features.common import Feature, Characteristic
+from capa.features.address import Address
+from capa.features.extractors.binexport2 import BinExport2Index, FunctionContext, BasicBlockContext, InstructionContext
+from capa.features.extractors.base_extractor import BBHandle, InsnHandle, FunctionHandle
+from capa.features.extractors.binexport2.helpers import (
+    BinExport2InstructionPatternMatcher,
+    mask_immediate,
+    is_address_mapped,
+    get_instruction_mnemonic,
+)
+from capa.features.extractors.binexport2.binexport2_pb2 import BinExport2
+from capa.features.extractors.binexport2.arch.intel.helpers import SECURITY_COOKIE_BYTES_DELTA
+
+logger = logging.getLogger(__name__)
+
+
+IGNORE_NUMBER_PATTERNS = BinExport2InstructionPatternMatcher.from_str(
+    """
+    ret  #int
+    retn #int
+    add  reg(stack), #int
+    sub  reg(stack), #int
+    """
+)
+
+NUMBER_PATTERNS = BinExport2InstructionPatternMatcher.from_str(
+    """
+    push #int0            ; capture #int0
+
+    # its a little tedious to enumerate all the address forms
+    # but at least we are explicit
+    cmp|and|or|test|mov|add|adc|sub|shl|shr|sal|sar  reg,                      #int0  ; capture #int0
+    cmp|and|or|test|mov|add|adc|sub|shl|shr|sal|sar [reg],                     #int0  ; capture #int0
+    cmp|and|or|test|mov|add|adc|sub|shl|shr|sal|sar [#int],                    #int0  ; capture #int0
+    cmp|and|or|test|mov|add|adc|sub|shl|shr|sal|sar [reg + #int],              #int0  ; capture #int0
+    cmp|and|or|test|mov|add|adc|sub|shl|shr|sal|sar [reg + reg + #int],        #int0  ; capture #int0
+    cmp|and|or|test|mov|add|adc|sub|shl|shr|sal|sar [reg + reg * #int],        #int0  ; capture #int0
+    cmp|and|or|test|mov|add|adc|sub|shl|shr|sal|sar [reg + reg * #int + #int], #int0  ; capture #int0
+
+    imul reg, reg, #int  ; capture #int
+    # note that int is first
+    cmp|test #int0, reg   ; capture #int0
+
+    # imagine reg is zero'd out, then this is like `mov reg, #int`
+    # which is not uncommon.
+    lea reg, [reg + #int]  ; capture #int
+    """
+)
+
+
+def extract_insn_number_features(
+    fh: FunctionHandle, _bbh: BBHandle, ih: InsnHandle
+) -> Iterator[tuple[Feature, Address]]:
+    fhi: FunctionContext = fh.inner
+    ii: InstructionContext = ih.inner
+
+    be2: BinExport2 = fhi.ctx.be2
+
+    if IGNORE_NUMBER_PATTERNS.match_with_be2(be2, ii.instruction_index):
+        return
+
+    match = NUMBER_PATTERNS.match_with_be2(be2, ii.instruction_index)
+    if not match:
+        return
+
+    value: int = mask_immediate(fhi.arch, match.expression.immediate)
+    if is_address_mapped(be2, value):
+        return
+
+    yield Number(value), ih.address
+    yield OperandNumber(match.operand_index, value), ih.address
+
+    instruction_index: int = ii.instruction_index
+    instruction: BinExport2.Instruction = be2.instruction[instruction_index]
+
+    mnemonic: str = get_instruction_mnemonic(be2, instruction)
+    if mnemonic.startswith("add"):
+        if 0 < value < MAX_STRUCTURE_SIZE:
+            yield Offset(value), ih.address
+            yield OperandOffset(match.operand_index, value), ih.address
+
+
+OFFSET_PATTERNS = BinExport2InstructionPatternMatcher.from_str(
+    """
+    mov|movzx|movsb|cmp [reg            +  reg * #int + #int0], #int  ; capture #int0
+    mov|movzx|movsb|cmp [reg            * #int + #int0],        #int  ; capture #int0
+    mov|movzx|movsb|cmp [reg            +  reg + #int0],        #int  ; capture #int0
+    mov|movzx|movsb|cmp [reg(not-stack) + #int0],               #int  ; capture #int0
+    mov|movzx|movsb|cmp [reg            +  reg * #int + #int0], reg   ; capture #int0
+    mov|movzx|movsb|cmp [reg            * #int + #int0],        reg   ; capture #int0
+    mov|movzx|movsb|cmp [reg            +  reg + #int0],        reg   ; capture #int0
+    mov|movzx|movsb|cmp [reg(not-stack) + #int0],               reg   ; capture #int0
+    mov|movzx|movsb|cmp|lea reg, [reg            +  reg * #int + #int0]  ; capture #int0
+    mov|movzx|movsb|cmp|lea reg, [reg            * #int + #int0]         ; capture #int0
+    mov|movzx|movsb|cmp|lea reg, [reg            +  reg + #int0]         ; capture #int0
+    mov|movzx|movsb|cmp|lea reg, [reg(not-stack) + #int0]                ; capture #int0
+    """
+)
+
+# these are patterns that access offset 0 from some pointer
+# (pointer is not the stack pointer).
+OFFSET_ZERO_PATTERNS = BinExport2InstructionPatternMatcher.from_str(
+    """
+    mov|movzx|movsb [reg(not-stack)], reg
+    mov|movzx|movsb [reg(not-stack)], #int
+    lea             reg,              [reg(not-stack)]
+    """
+)
+
+
+def extract_insn_offset_features(
+    fh: FunctionHandle, bbh: BBHandle, ih: InsnHandle
+) -> Iterator[tuple[Feature, Address]]:
+    fhi: FunctionContext = fh.inner
+    ii: InstructionContext = ih.inner
+
+    be2: BinExport2 = fhi.ctx.be2
+
+    match = OFFSET_PATTERNS.match_with_be2(be2, ii.instruction_index)
+    if not match:
+        match = OFFSET_ZERO_PATTERNS.match_with_be2(be2, ii.instruction_index)
+        if not match:
+            return
+
+        yield Offset(0), ih.address
+        yield OperandOffset(match.operand_index, 0), ih.address
+
+    value = mask_immediate(fhi.arch, match.expression.immediate)
+    if is_address_mapped(be2, value):
+        return
+
+    value = capa.features.extractors.binexport2.helpers.twos_complement(fhi.arch, value, 32)
+    yield Offset(value), ih.address
+    yield OperandOffset(match.operand_index, value), ih.address
+
+
+def is_security_cookie(
+    fhi: FunctionContext,
+    bbi: BasicBlockContext,
+    instruction_address: int,
+    instruction: BinExport2.Instruction,
+) -> bool:
+    """
+    check if an instruction is related to security cookie checks.
+    """
+    be2: BinExport2 = fhi.ctx.be2
+    idx: BinExport2Index = fhi.ctx.idx
+
+    # security cookie check should use SP or BP
+    op1: BinExport2.Operand = be2.operand[instruction.operand_index[1]]
+    op1_exprs: list[BinExport2.Expression] = [be2.expression[expr_i] for expr_i in op1.expression_index]
+    if all(expr.symbol.lower() not in ("bp", "esp", "ebp", "rbp", "rsp") for expr in op1_exprs):
+        return False
+
+    # check_nzxor_security_cookie_delta
+    # if insn falls at the start of first entry block of the parent function.
+    flow_graph: BinExport2.FlowGraph = be2.flow_graph[fhi.flow_graph_index]
+    basic_block_index: int = bbi.basic_block_index
+    bb: BinExport2.BasicBlock = be2.basic_block[basic_block_index]
+    if flow_graph.entry_basic_block_index == basic_block_index:
+        first_addr: int = min((idx.insn_address_by_index[ir.begin_index] for ir in bb.instruction_index))
+        if instruction_address < first_addr + SECURITY_COOKIE_BYTES_DELTA:
+            return True
+    # or insn falls at the end before return in a terminal basic block.
+    if basic_block_index not in (e.source_basic_block_index for e in flow_graph.edge):
+        last_addr: int = max((idx.insn_address_by_index[ir.end_index - 1] for ir in bb.instruction_index))
+        if instruction_address > last_addr - SECURITY_COOKIE_BYTES_DELTA:
+            return True
+    return False
+
+
+NZXOR_PATTERNS = BinExport2InstructionPatternMatcher.from_str(
+    """
+    xor|xorpd|xorps|pxor reg, reg
+    xor|xorpd|xorps|pxor reg, #int
+    """
+)
+
+
+def extract_insn_nzxor_characteristic_features(
+    fh: FunctionHandle, bbh: BBHandle, ih: InsnHandle
+) -> Iterator[tuple[Feature, Address]]:
+    """
+    parse non-zeroing XOR instruction from the given instruction.
+    ignore expected non-zeroing XORs, e.g. security cookies.
+    """
+    fhi: FunctionContext = fh.inner
+    ii: InstructionContext = ih.inner
+
+    be2: BinExport2 = fhi.ctx.be2
+    idx: BinExport2Index = fhi.ctx.idx
+
+    if NZXOR_PATTERNS.match_with_be2(be2, ii.instruction_index) is None:
+        return
+
+    instruction: BinExport2.Instruction = be2.instruction[ii.instruction_index]
+    # guaranteed to be simple int/reg operands
+    # so we don't have to realize the tree/list.
+    operands: list[BinExport2.Operand] = [be2.operand[operand_index] for operand_index in instruction.operand_index]
+
+    if operands[0] == operands[1]:
+        return
+
+    instruction_address: int = idx.insn_address_by_index[ii.instruction_index]
+    if is_security_cookie(fhi, bbh.inner, instruction_address, instruction):
+        return
+
+    yield Characteristic("nzxor"), ih.address
+
+
+INDIRECT_CALL_PATTERNS = BinExport2InstructionPatternMatcher.from_str(
+    """
+    call|jmp reg0
+    call|jmp [reg + reg * #int + #int]
+    call|jmp [reg + reg * #int]
+    call|jmp [reg * #int + #int]
+    call|jmp [reg + reg + #int]
+    call|jmp [reg + #int]
+    call|jmp [reg]
+    """
+)
+
+
+def extract_function_indirect_call_characteristic_features(
+    fh: FunctionHandle, bbh: BBHandle, ih: InsnHandle
+) -> Iterator[tuple[Feature, Address]]:
+    fhi: FunctionContext = fh.inner
+    ii: InstructionContext = ih.inner
+    be2: BinExport2 = fhi.ctx.be2
+
+    match = INDIRECT_CALL_PATTERNS.match_with_be2(be2, ii.instruction_index)
+    if match is None:
+        return
+
+    yield Characteristic("indirect call"), ih.address

capa/features/extractors/binexport2/basicblock.py

@@ -0,0 +1,47 @@
+# Copyright 2023 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+from typing import Iterator
+
+from capa.features.common import Feature, Characteristic
+from capa.features.address import Address, AbsoluteVirtualAddress
+from capa.features.basicblock import BasicBlock
+from capa.features.extractors.binexport2 import FunctionContext, BasicBlockContext
+from capa.features.extractors.base_extractor import BBHandle, FunctionHandle
+from capa.features.extractors.binexport2.binexport2_pb2 import BinExport2
+
+
+def extract_bb_tight_loop(fh: FunctionHandle, bbh: BBHandle) -> Iterator[tuple[Feature, Address]]:
+    fhi: FunctionContext = fh.inner
+    bbi: BasicBlockContext = bbh.inner
+
+    idx = fhi.ctx.idx
+
+    basic_block_index: int = bbi.basic_block_index
+    target_edges: list[BinExport2.FlowGraph.Edge] = idx.target_edges_by_basic_block_index[basic_block_index]
+    if basic_block_index in (e.source_basic_block_index for e in target_edges):
+        basic_block_address: int = idx.get_basic_block_address(basic_block_index)
+        yield Characteristic("tight loop"), AbsoluteVirtualAddress(basic_block_address)
+
+
+def extract_features(fh: FunctionHandle, bbh: BBHandle) -> Iterator[tuple[Feature, Address]]:
+    """extract basic block features"""
+    for bb_handler in BASIC_BLOCK_HANDLERS:
+        for feature, addr in bb_handler(fh, bbh):
+            yield feature, addr
+    yield BasicBlock(), bbh.address
+
+
+BASIC_BLOCK_HANDLERS = (extract_bb_tight_loop,)

capa/features/extractors/binexport2/binexport2_pb2.pyi

@@ -0,0 +1,784 @@
+"""
+@generated by mypy-protobuf.  Do not edit manually!
+isort:skip_file
+The representation is generic to accommodate various source architectures.
+In particular 32 and 64 bit versions of x86, ARM, PowerPC and MIPS have been
+tested.
+
+Multiple levels of deduping have been applied to make the format more compact
+and avoid redundant data duplication. Some of this due to hard-earned
+experience trying to cope with intentionally obfuscated malicious binaries.
+Note in particular that the same instruction may occur in multiple basic
+blocks and the same basic block in multiple functions (instruction and basic
+block sharing). Implemented naively, malware can use this to cause
+combinatorial explosion in memory usage, DOSing the analyst. This format
+should store every unique expression, mnemonic, operand, instruction and
+basic block only once instead of duplicating the information for every
+instance of it.
+
+This format does _not_ try to be 100% backwards compatible with the old
+version. In particular, we do not store IDA's comment types, making lossless
+porting of IDA comments impossible. We do however, store comments and
+expression substitutions, so porting the actual data is possible, just not
+the exact IDA type.
+
+While it would be more natural to use addresses when defining call graph and
+flow graph edges and other such references, it is more efficient to employ
+one more level of indirection and use indices into the basic block or
+function arrays instead. This is because addresses will usually use most of
+the available 64 bit space while indices will be much smaller and compress
+much better (less randomly distributed).
+
+We omit all fields that are set to their default value anyways. Note that
+this has two side effects:
+  - changing the defaults in this proto file will, in effect, change what's
+    read from disk
+  - the generated code has_* methods are somewhat less useful
+WARNING: We omit the defaults manually in the code writing the data. Do not
+         change the defaults here without changing the code!
+
+TODO(cblichmann): Link flow graphs to call graph nodes. The connection is
+                  there via the address, but tricky to extract.
+"""
+import builtins
+import collections.abc
+import google.protobuf.descriptor
+import google.protobuf.internal.containers
+import google.protobuf.internal.enum_type_wrapper
+import google.protobuf.message
+import sys
+import typing
+
+if sys.version_info >= (3, 10):
+    import typing as typing_extensions
+else:
+    import typing_extensions
+
+DESCRIPTOR: google.protobuf.descriptor.FileDescriptor
+
+@typing_extensions.final
+class BinExport2(google.protobuf.message.Message):
+    DESCRIPTOR: google.protobuf.descriptor.Descriptor
+
+    @typing_extensions.final
+    class Meta(google.protobuf.message.Message):
+        DESCRIPTOR: google.protobuf.descriptor.Descriptor
+
+        EXECUTABLE_NAME_FIELD_NUMBER: builtins.int
+        EXECUTABLE_ID_FIELD_NUMBER: builtins.int
+        ARCHITECTURE_NAME_FIELD_NUMBER: builtins.int
+        TIMESTAMP_FIELD_NUMBER: builtins.int
+        executable_name: builtins.str
+        """Input binary filename including file extension but excluding file path.
+        example: "insider_gcc.exe"
+        """
+        executable_id: builtins.str
+        """Application defined executable id. Often the SHA256 hash of the input
+        binary.
+        """
+        architecture_name: builtins.str
+        """Input architecture name, e.g. x86-32."""
+        timestamp: builtins.int
+        """When did this file get created? Unix time. This may be used for some
+        primitive versioning in case the file format ever changes.
+        """
+        def __init__(
+            self,
+            *,
+            executable_name: builtins.str | None = ...,
+            executable_id: builtins.str | None = ...,
+            architecture_name: builtins.str | None = ...,
+            timestamp: builtins.int | None = ...,
+        ) -> None: ...
+        def HasField(self, field_name: typing_extensions.Literal["architecture_name", b"architecture_name", "executable_id", b"executable_id", "executable_name", b"executable_name", "timestamp", b"timestamp"]) -> builtins.bool: ...
+        def ClearField(self, field_name: typing_extensions.Literal["architecture_name", b"architecture_name", "executable_id", b"executable_id", "executable_name", b"executable_name", "timestamp", b"timestamp"]) -> None: ...
+
+    @typing_extensions.final
+    class CallGraph(google.protobuf.message.Message):
+        DESCRIPTOR: google.protobuf.descriptor.Descriptor
+
+        @typing_extensions.final
+        class Vertex(google.protobuf.message.Message):
+            DESCRIPTOR: google.protobuf.descriptor.Descriptor
+
+            class _Type:
+                ValueType = typing.NewType("ValueType", builtins.int)
+                V: typing_extensions.TypeAlias = ValueType
+
+            class _TypeEnumTypeWrapper(google.protobuf.internal.enum_type_wrapper._EnumTypeWrapper[BinExport2.CallGraph.Vertex._Type.ValueType], builtins.type):
+                DESCRIPTOR: google.protobuf.descriptor.EnumDescriptor
+                NORMAL: BinExport2.CallGraph.Vertex._Type.ValueType  # 0
+                """Regular function with full disassembly."""
+                LIBRARY: BinExport2.CallGraph.Vertex._Type.ValueType  # 1
+                """This function is a well known library function."""
+                IMPORTED: BinExport2.CallGraph.Vertex._Type.ValueType  # 2
+                """Imported from a dynamic link library (e.g. dll)."""
+                THUNK: BinExport2.CallGraph.Vertex._Type.ValueType  # 3
+                """A thunk function, forwarding its work via an unconditional jump."""
+                INVALID: BinExport2.CallGraph.Vertex._Type.ValueType  # 4
+                """An invalid function (a function that contained invalid code or was
+                considered invalid by some heuristics).
+                """
+
+            class Type(_Type, metaclass=_TypeEnumTypeWrapper): ...
+            NORMAL: BinExport2.CallGraph.Vertex.Type.ValueType  # 0
+            """Regular function with full disassembly."""
+            LIBRARY: BinExport2.CallGraph.Vertex.Type.ValueType  # 1
+            """This function is a well known library function."""
+            IMPORTED: BinExport2.CallGraph.Vertex.Type.ValueType  # 2
+            """Imported from a dynamic link library (e.g. dll)."""
+            THUNK: BinExport2.CallGraph.Vertex.Type.ValueType  # 3
+            """A thunk function, forwarding its work via an unconditional jump."""
+            INVALID: BinExport2.CallGraph.Vertex.Type.ValueType  # 4
+            """An invalid function (a function that contained invalid code or was
+            considered invalid by some heuristics).
+            """
+
+            ADDRESS_FIELD_NUMBER: builtins.int
+            TYPE_FIELD_NUMBER: builtins.int
+            MANGLED_NAME_FIELD_NUMBER: builtins.int
+            DEMANGLED_NAME_FIELD_NUMBER: builtins.int
+            LIBRARY_INDEX_FIELD_NUMBER: builtins.int
+            MODULE_INDEX_FIELD_NUMBER: builtins.int
+            address: builtins.int
+            """The function's entry point address. Messages need to be sorted, see
+            comment below on `vertex`.
+            """
+            type: global___BinExport2.CallGraph.Vertex.Type.ValueType
+            mangled_name: builtins.str
+            """If the function has a user defined, real name it will be given here.
+            main() is a proper name, sub_BAADF00D is not (auto generated dummy
+            name).
+            """
+            demangled_name: builtins.str
+            """Demangled name if the function is a mangled C++ function and we could
+            demangle it.
+            """
+            library_index: builtins.int
+            """If this is a library function, what is its index in library arrays."""
+            module_index: builtins.int
+            """If module name, such as class name for DEX files, is present - index in
+            module table.
+            """
+            def __init__(
+                self,
+                *,
+                address: builtins.int | None = ...,
+                type: global___BinExport2.CallGraph.Vertex.Type.ValueType | None = ...,
+                mangled_name: builtins.str | None = ...,
+                demangled_name: builtins.str | None = ...,
+                library_index: builtins.int | None = ...,
+                module_index: builtins.int | None = ...,
+            ) -> None: ...
+            def HasField(self, field_name: typing_extensions.Literal["address", b"address", "demangled_name", b"demangled_name", "library_index", b"library_index", "mangled_name", b"mangled_name", "module_index", b"module_index", "type", b"type"]) -> builtins.bool: ...
+            def ClearField(self, field_name: typing_extensions.Literal["address", b"address", "demangled_name", b"demangled_name", "library_index", b"library_index", "mangled_name", b"mangled_name", "module_index", b"module_index", "type", b"type"]) -> None: ...
+
+        @typing_extensions.final
+        class Edge(google.protobuf.message.Message):
+            DESCRIPTOR: google.protobuf.descriptor.Descriptor
+
+            SOURCE_VERTEX_INDEX_FIELD_NUMBER: builtins.int
+            TARGET_VERTEX_INDEX_FIELD_NUMBER: builtins.int
+            source_vertex_index: builtins.int
+            """source and target index into the vertex repeated field."""
+            target_vertex_index: builtins.int
+            def __init__(
+                self,
+                *,
+                source_vertex_index: builtins.int | None = ...,
+                target_vertex_index: builtins.int | None = ...,
+            ) -> None: ...
+            def HasField(self, field_name: typing_extensions.Literal["source_vertex_index", b"source_vertex_index", "target_vertex_index", b"target_vertex_index"]) -> builtins.bool: ...
+            def ClearField(self, field_name: typing_extensions.Literal["source_vertex_index", b"source_vertex_index", "target_vertex_index", b"target_vertex_index"]) -> None: ...
+
+        VERTEX_FIELD_NUMBER: builtins.int
+        EDGE_FIELD_NUMBER: builtins.int
+        @property
+        def vertex(self) -> google.protobuf.internal.containers.RepeatedCompositeFieldContainer[global___BinExport2.CallGraph.Vertex]:
+            """vertices == functions in the call graph.
+            Important: Most downstream tooling (notably BinDiff), need these to be
+                       sorted by `Vertex::address` (ascending). For C++, the
+                       `BinExport2Writer` class enforces this invariant.
+            """
+        @property
+        def edge(self) -> google.protobuf.internal.containers.RepeatedCompositeFieldContainer[global___BinExport2.CallGraph.Edge]:
+            """edges == calls in the call graph."""
+        def __init__(
+            self,
+            *,
+            vertex: collections.abc.Iterable[global___BinExport2.CallGraph.Vertex] | None = ...,
+            edge: collections.abc.Iterable[global___BinExport2.CallGraph.Edge] | None = ...,
+        ) -> None: ...
+        def ClearField(self, field_name: typing_extensions.Literal["edge", b"edge", "vertex", b"vertex"]) -> None: ...
+
+    @typing_extensions.final
+    class Expression(google.protobuf.message.Message):
+        """An operand consists of 1 or more expressions, linked together as a tree."""
+
+        DESCRIPTOR: google.protobuf.descriptor.Descriptor
+
+        class _Type:
+            ValueType = typing.NewType("ValueType", builtins.int)
+            V: typing_extensions.TypeAlias = ValueType
+
+        class _TypeEnumTypeWrapper(google.protobuf.internal.enum_type_wrapper._EnumTypeWrapper[BinExport2.Expression._Type.ValueType], builtins.type):
+            DESCRIPTOR: google.protobuf.descriptor.EnumDescriptor
+            SYMBOL: BinExport2.Expression._Type.ValueType  # 1
+            IMMEDIATE_INT: BinExport2.Expression._Type.ValueType  # 2
+            IMMEDIATE_FLOAT: BinExport2.Expression._Type.ValueType  # 3
+            OPERATOR: BinExport2.Expression._Type.ValueType  # 4
+            REGISTER: BinExport2.Expression._Type.ValueType  # 5
+            SIZE_PREFIX: BinExport2.Expression._Type.ValueType  # 6
+            DEREFERENCE: BinExport2.Expression._Type.ValueType  # 7
+
+        class Type(_Type, metaclass=_TypeEnumTypeWrapper): ...
+        SYMBOL: BinExport2.Expression.Type.ValueType  # 1
+        IMMEDIATE_INT: BinExport2.Expression.Type.ValueType  # 2
+        IMMEDIATE_FLOAT: BinExport2.Expression.Type.ValueType  # 3
+        OPERATOR: BinExport2.Expression.Type.ValueType  # 4
+        REGISTER: BinExport2.Expression.Type.ValueType  # 5
+        SIZE_PREFIX: BinExport2.Expression.Type.ValueType  # 6
+        DEREFERENCE: BinExport2.Expression.Type.ValueType  # 7
+
+        TYPE_FIELD_NUMBER: builtins.int
+        SYMBOL_FIELD_NUMBER: builtins.int
+        IMMEDIATE_FIELD_NUMBER: builtins.int
+        PARENT_INDEX_FIELD_NUMBER: builtins.int
+        IS_RELOCATION_FIELD_NUMBER: builtins.int
+        type: global___BinExport2.Expression.Type.ValueType
+        """IMMEDIATE_INT is by far the most common type and thus we can save some
+        space by omitting it as the default.
+        """
+        symbol: builtins.str
+        """Symbol for this expression. Interpretation depends on type. Examples
+        include: "eax", "[", "+"
+        """
+        immediate: builtins.int
+        """If the expression can be interpreted as an integer value (IMMEDIATE_INT)
+        the value is given here.
+        """
+        parent_index: builtins.int
+        """The parent expression. Example expression tree for the second operand of:
+        mov eax, b4 [ebx + 12]
+        "b4" --- "[" --- "+" --- "ebx"
+                              \\  "12"
+        """
+        is_relocation: builtins.bool
+        """true if the expression has entry in relocation table"""
+        def __init__(
+            self,
+            *,
+            type: global___BinExport2.Expression.Type.ValueType | None = ...,
+            symbol: builtins.str | None = ...,
+            immediate: builtins.int | None = ...,
+            parent_index: builtins.int | None = ...,
+            is_relocation: builtins.bool | None = ...,
+        ) -> None: ...
+        def HasField(self, field_name: typing_extensions.Literal["immediate", b"immediate", "is_relocation", b"is_relocation", "parent_index", b"parent_index", "symbol", b"symbol", "type", b"type"]) -> builtins.bool: ...
+        def ClearField(self, field_name: typing_extensions.Literal["immediate", b"immediate", "is_relocation", b"is_relocation", "parent_index", b"parent_index", "symbol", b"symbol", "type", b"type"]) -> None: ...
+
+    @typing_extensions.final
+    class Operand(google.protobuf.message.Message):
+        """An instruction may have 0 or more operands."""
+
+        DESCRIPTOR: google.protobuf.descriptor.Descriptor
+
+        EXPRESSION_INDEX_FIELD_NUMBER: builtins.int
+        @property
+        def expression_index(self) -> google.protobuf.internal.containers.RepeatedScalarFieldContainer[builtins.int]:
+            """Contains all expressions constituting this operand. All expressions
+            should be linked into a single tree, i.e. there should only be one
+            expression in this list with parent_index == NULL and all others should
+            descend from that. Rendering order for expressions on the same tree level
+            (siblings) is implicitly given by the order they are referenced in this
+            repeated field.
+            Implicit: expression sequence
+            """
+        def __init__(
+            self,
+            *,
+            expression_index: collections.abc.Iterable[builtins.int] | None = ...,
+        ) -> None: ...
+        def ClearField(self, field_name: typing_extensions.Literal["expression_index", b"expression_index"]) -> None: ...
+
+    @typing_extensions.final
+    class Mnemonic(google.protobuf.message.Message):
+        """An instruction has exactly 1 mnemonic."""
+
+        DESCRIPTOR: google.protobuf.descriptor.Descriptor
+
+        NAME_FIELD_NUMBER: builtins.int
+        name: builtins.str
+        """Literal representation of the mnemonic, e.g.: "mov"."""
+        def __init__(
+            self,
+            *,
+            name: builtins.str | None = ...,
+        ) -> None: ...
+        def HasField(self, field_name: typing_extensions.Literal["name", b"name"]) -> builtins.bool: ...
+        def ClearField(self, field_name: typing_extensions.Literal["name", b"name"]) -> None: ...
+
+    @typing_extensions.final
+    class Instruction(google.protobuf.message.Message):
+        DESCRIPTOR: google.protobuf.descriptor.Descriptor
+
+        ADDRESS_FIELD_NUMBER: builtins.int
+        CALL_TARGET_FIELD_NUMBER: builtins.int
+        MNEMONIC_INDEX_FIELD_NUMBER: builtins.int
+        OPERAND_INDEX_FIELD_NUMBER: builtins.int
+        RAW_BYTES_FIELD_NUMBER: builtins.int
+        COMMENT_INDEX_FIELD_NUMBER: builtins.int
+        address: builtins.int
+        """This will only be filled for instructions that do not just flow from the
+        immediately preceding instruction. Regular instructions will have to
+        calculate their own address by adding raw_bytes.size() to the previous
+        instruction's address.
+        """
+        @property
+        def call_target(self) -> google.protobuf.internal.containers.RepeatedScalarFieldContainer[builtins.int]:
+            """If this is a call instruction and call targets could be determined
+            they'll be given here. Note that we may or may not have a flow graph for
+            the target and thus cannot use an index into the flow graph table here.
+            We could potentially use call graph nodes, but linking instructions to
+            the call graph directly does not seem a good choice.
+            """
+        mnemonic_index: builtins.int
+        """Index into the mnemonic array of strings. Used for de-duping the data.
+        The default value is used for the most common mnemonic in the executable.
+        """
+        @property
+        def operand_index(self) -> google.protobuf.internal.containers.RepeatedScalarFieldContainer[builtins.int]:
+            """Indices into the operand tree. On X86 this can be 0, 1 or 2 elements
+            long, 3 elements with VEX/EVEX.
+            Implicit: operand sequence
+            """
+        raw_bytes: builtins.bytes
+        """The unmodified input bytes corresponding to this instruction."""
+        @property
+        def comment_index(self) -> google.protobuf.internal.containers.RepeatedScalarFieldContainer[builtins.int]:
+            """Implicit: comment sequence"""
+        def __init__(
+            self,
+            *,
+            address: builtins.int | None = ...,
+            call_target: collections.abc.Iterable[builtins.int] | None = ...,
+            mnemonic_index: builtins.int | None = ...,
+            operand_index: collections.abc.Iterable[builtins.int] | None = ...,
+            raw_bytes: builtins.bytes | None = ...,
+            comment_index: collections.abc.Iterable[builtins.int] | None = ...,
+        ) -> None: ...
+        def HasField(self, field_name: typing_extensions.Literal["address", b"address", "mnemonic_index", b"mnemonic_index", "raw_bytes", b"raw_bytes"]) -> builtins.bool: ...
+        def ClearField(self, field_name: typing_extensions.Literal["address", b"address", "call_target", b"call_target", "comment_index", b"comment_index", "mnemonic_index", b"mnemonic_index", "operand_index", b"operand_index", "raw_bytes", b"raw_bytes"]) -> None: ...
+
+    @typing_extensions.final
+    class BasicBlock(google.protobuf.message.Message):
+        DESCRIPTOR: google.protobuf.descriptor.Descriptor
+
+        @typing_extensions.final
+        class IndexRange(google.protobuf.message.Message):
+            """This is a space optimization. The instructions for an individual basic
+            block will usually be in a continuous index range. Thus it is more
+            efficient to store the range instead of individual indices. However, this
+            does not hold true for all basic blocks, so we need to be able to store
+            multiple index ranges per block.
+            """
+
+            DESCRIPTOR: google.protobuf.descriptor.Descriptor
+
+            BEGIN_INDEX_FIELD_NUMBER: builtins.int
+            END_INDEX_FIELD_NUMBER: builtins.int
+            begin_index: builtins.int
+            """These work like begin and end iterators, i.e. the sequence is
+            [begin_index, end_index). If the sequence only contains a single
+            element end_index will be omitted.
+            """
+            end_index: builtins.int
+            def __init__(
+                self,
+                *,
+                begin_index: builtins.int | None = ...,
+                end_index: builtins.int | None = ...,
+            ) -> None: ...
+            def HasField(self, field_name: typing_extensions.Literal["begin_index", b"begin_index", "end_index", b"end_index"]) -> builtins.bool: ...
+            def ClearField(self, field_name: typing_extensions.Literal["begin_index", b"begin_index", "end_index", b"end_index"]) -> None: ...
+
+        INSTRUCTION_INDEX_FIELD_NUMBER: builtins.int
+        @property
+        def instruction_index(self) -> google.protobuf.internal.containers.RepeatedCompositeFieldContainer[global___BinExport2.BasicBlock.IndexRange]:
+            """Implicit: instruction sequence"""
+        def __init__(
+            self,
+            *,
+            instruction_index: collections.abc.Iterable[global___BinExport2.BasicBlock.IndexRange] | None = ...,
+        ) -> None: ...
+        def ClearField(self, field_name: typing_extensions.Literal["instruction_index", b"instruction_index"]) -> None: ...
+
+    @typing_extensions.final
+    class FlowGraph(google.protobuf.message.Message):
+        DESCRIPTOR: google.protobuf.descriptor.Descriptor
+
+        @typing_extensions.final
+        class Edge(google.protobuf.message.Message):
+            DESCRIPTOR: google.protobuf.descriptor.Descriptor
+
+            class _Type:
+                ValueType = typing.NewType("ValueType", builtins.int)
+                V: typing_extensions.TypeAlias = ValueType
+
+            class _TypeEnumTypeWrapper(google.protobuf.internal.enum_type_wrapper._EnumTypeWrapper[BinExport2.FlowGraph.Edge._Type.ValueType], builtins.type):
+                DESCRIPTOR: google.protobuf.descriptor.EnumDescriptor
+                CONDITION_TRUE: BinExport2.FlowGraph.Edge._Type.ValueType  # 1
+                CONDITION_FALSE: BinExport2.FlowGraph.Edge._Type.ValueType  # 2
+                UNCONDITIONAL: BinExport2.FlowGraph.Edge._Type.ValueType  # 3
+                SWITCH: BinExport2.FlowGraph.Edge._Type.ValueType  # 4
+
+            class Type(_Type, metaclass=_TypeEnumTypeWrapper): ...
+            CONDITION_TRUE: BinExport2.FlowGraph.Edge.Type.ValueType  # 1
+            CONDITION_FALSE: BinExport2.FlowGraph.Edge.Type.ValueType  # 2
+            UNCONDITIONAL: BinExport2.FlowGraph.Edge.Type.ValueType  # 3
+            SWITCH: BinExport2.FlowGraph.Edge.Type.ValueType  # 4
+
+            SOURCE_BASIC_BLOCK_INDEX_FIELD_NUMBER: builtins.int
+            TARGET_BASIC_BLOCK_INDEX_FIELD_NUMBER: builtins.int
+            TYPE_FIELD_NUMBER: builtins.int
+            IS_BACK_EDGE_FIELD_NUMBER: builtins.int
+            source_basic_block_index: builtins.int
+            """Source instruction will always be the last instruction of the source
+            basic block, target instruction the first instruction of the target
+            basic block.
+            """
+            target_basic_block_index: builtins.int
+            type: global___BinExport2.FlowGraph.Edge.Type.ValueType
+            is_back_edge: builtins.bool
+            """Indicates whether this is a loop edge as determined by Lengauer-Tarjan."""
+            def __init__(
+                self,
+                *,
+                source_basic_block_index: builtins.int | None = ...,
+                target_basic_block_index: builtins.int | None = ...,
+                type: global___BinExport2.FlowGraph.Edge.Type.ValueType | None = ...,
+                is_back_edge: builtins.bool | None = ...,
+            ) -> None: ...
+            def HasField(self, field_name: typing_extensions.Literal["is_back_edge", b"is_back_edge", "source_basic_block_index", b"source_basic_block_index", "target_basic_block_index", b"target_basic_block_index", "type", b"type"]) -> builtins.bool: ...
+            def ClearField(self, field_name: typing_extensions.Literal["is_back_edge", b"is_back_edge", "source_basic_block_index", b"source_basic_block_index", "target_basic_block_index", b"target_basic_block_index", "type", b"type"]) -> None: ...
+
+        BASIC_BLOCK_INDEX_FIELD_NUMBER: builtins.int
+        ENTRY_BASIC_BLOCK_INDEX_FIELD_NUMBER: builtins.int
+        EDGE_FIELD_NUMBER: builtins.int
+        @property
+        def basic_block_index(self) -> google.protobuf.internal.containers.RepeatedScalarFieldContainer[builtins.int]:
+            """Basic blocks are sorted by address."""
+        entry_basic_block_index: builtins.int
+        """The flow graph's entry point address is the first instruction of the
+        entry_basic_block.
+        """
+        @property
+        def edge(self) -> google.protobuf.internal.containers.RepeatedCompositeFieldContainer[global___BinExport2.FlowGraph.Edge]: ...
+        def __init__(
+            self,
+            *,
+            basic_block_index: collections.abc.Iterable[builtins.int] | None = ...,
+            entry_basic_block_index: builtins.int | None = ...,
+            edge: collections.abc.Iterable[global___BinExport2.FlowGraph.Edge] | None = ...,
+        ) -> None: ...
+        def HasField(self, field_name: typing_extensions.Literal["entry_basic_block_index", b"entry_basic_block_index"]) -> builtins.bool: ...
+        def ClearField(self, field_name: typing_extensions.Literal["basic_block_index", b"basic_block_index", "edge", b"edge", "entry_basic_block_index", b"entry_basic_block_index"]) -> None: ...
+
+    @typing_extensions.final
+    class Reference(google.protobuf.message.Message):
+        """Generic reference class used for address comments (deprecated), string
+        references and expression substitutions. It allows referencing from an
+        instruction, operand, expression subtree tuple to a de-duped string in the
+        string table.
+        """
+
+        DESCRIPTOR: google.protobuf.descriptor.Descriptor
+
+        INSTRUCTION_INDEX_FIELD_NUMBER: builtins.int
+        INSTRUCTION_OPERAND_INDEX_FIELD_NUMBER: builtins.int
+        OPERAND_EXPRESSION_INDEX_FIELD_NUMBER: builtins.int
+        STRING_TABLE_INDEX_FIELD_NUMBER: builtins.int
+        instruction_index: builtins.int
+        """Index into the global instruction table."""
+        instruction_operand_index: builtins.int
+        """Index into the operand array local to an instruction."""
+        operand_expression_index: builtins.int
+        """Index into the expression array local to an operand."""
+        string_table_index: builtins.int
+        """Index into the global string table."""
+        def __init__(
+            self,
+            *,
+            instruction_index: builtins.int | None = ...,
+            instruction_operand_index: builtins.int | None = ...,
+            operand_expression_index: builtins.int | None = ...,
+            string_table_index: builtins.int | None = ...,
+        ) -> None: ...
+        def HasField(self, field_name: typing_extensions.Literal["instruction_index", b"instruction_index", "instruction_operand_index", b"instruction_operand_index", "operand_expression_index", b"operand_expression_index", "string_table_index", b"string_table_index"]) -> builtins.bool: ...
+        def ClearField(self, field_name: typing_extensions.Literal["instruction_index", b"instruction_index", "instruction_operand_index", b"instruction_operand_index", "operand_expression_index", b"operand_expression_index", "string_table_index", b"string_table_index"]) -> None: ...
+
+    @typing_extensions.final
+    class DataReference(google.protobuf.message.Message):
+        DESCRIPTOR: google.protobuf.descriptor.Descriptor
+
+        INSTRUCTION_INDEX_FIELD_NUMBER: builtins.int
+        ADDRESS_FIELD_NUMBER: builtins.int
+        instruction_index: builtins.int
+        """Index into the global instruction table."""
+        address: builtins.int
+        """Address being referred."""
+        def __init__(
+            self,
+            *,
+            instruction_index: builtins.int | None = ...,
+            address: builtins.int | None = ...,
+        ) -> None: ...
+        def HasField(self, field_name: typing_extensions.Literal["address", b"address", "instruction_index", b"instruction_index"]) -> builtins.bool: ...
+        def ClearField(self, field_name: typing_extensions.Literal["address", b"address", "instruction_index", b"instruction_index"]) -> None: ...
+
+    @typing_extensions.final
+    class Comment(google.protobuf.message.Message):
+        DESCRIPTOR: google.protobuf.descriptor.Descriptor
+
+        class _Type:
+            ValueType = typing.NewType("ValueType", builtins.int)
+            V: typing_extensions.TypeAlias = ValueType
+
+        class _TypeEnumTypeWrapper(google.protobuf.internal.enum_type_wrapper._EnumTypeWrapper[BinExport2.Comment._Type.ValueType], builtins.type):
+            DESCRIPTOR: google.protobuf.descriptor.EnumDescriptor
+            DEFAULT: BinExport2.Comment._Type.ValueType  # 0
+            """A regular instruction comment. Typically displayed next to the
+            instruction disassembly.
+            """
+            ANTERIOR: BinExport2.Comment._Type.ValueType  # 1
+            """A comment line that is typically displayed before (above) the
+            instruction it refers to.
+            """
+            POSTERIOR: BinExport2.Comment._Type.ValueType  # 2
+            """Like ANTERIOR, but a typically displayed after (below)."""
+            FUNCTION: BinExport2.Comment._Type.ValueType  # 3
+            """Similar to an ANTERIOR comment, but applies to the beginning of an
+            identified function. Programs displaying the proto may choose to render
+            these differently (e.g. above an inferred function signature).
+            """
+            ENUM: BinExport2.Comment._Type.ValueType  # 4
+            """Named constants, bitfields and similar."""
+            LOCATION: BinExport2.Comment._Type.ValueType  # 5
+            """Named locations, usually the target of a jump."""
+            GLOBAL_REFERENCE: BinExport2.Comment._Type.ValueType  # 6
+            """Data cross references."""
+            LOCAL_REFERENCE: BinExport2.Comment._Type.ValueType  # 7
+            """Local/stack variables."""
+
+        class Type(_Type, metaclass=_TypeEnumTypeWrapper): ...
+        DEFAULT: BinExport2.Comment.Type.ValueType  # 0
+        """A regular instruction comment. Typically displayed next to the
+        instruction disassembly.
+        """
+        ANTERIOR: BinExport2.Comment.Type.ValueType  # 1
+        """A comment line that is typically displayed before (above) the
+        instruction it refers to.
+        """
+        POSTERIOR: BinExport2.Comment.Type.ValueType  # 2
+        """Like ANTERIOR, but a typically displayed after (below)."""
+        FUNCTION: BinExport2.Comment.Type.ValueType  # 3
+        """Similar to an ANTERIOR comment, but applies to the beginning of an
+        identified function. Programs displaying the proto may choose to render
+        these differently (e.g. above an inferred function signature).
+        """
+        ENUM: BinExport2.Comment.Type.ValueType  # 4
+        """Named constants, bitfields and similar."""
+        LOCATION: BinExport2.Comment.Type.ValueType  # 5
+        """Named locations, usually the target of a jump."""
+        GLOBAL_REFERENCE: BinExport2.Comment.Type.ValueType  # 6
+        """Data cross references."""
+        LOCAL_REFERENCE: BinExport2.Comment.Type.ValueType  # 7
+        """Local/stack variables."""
+
+        INSTRUCTION_INDEX_FIELD_NUMBER: builtins.int
+        INSTRUCTION_OPERAND_INDEX_FIELD_NUMBER: builtins.int
+        OPERAND_EXPRESSION_INDEX_FIELD_NUMBER: builtins.int
+        STRING_TABLE_INDEX_FIELD_NUMBER: builtins.int
+        REPEATABLE_FIELD_NUMBER: builtins.int
+        TYPE_FIELD_NUMBER: builtins.int
+        instruction_index: builtins.int
+        """Index into the global instruction table. This is here to enable
+        comment processing without having to iterate over all instructions.
+        There is an N:M mapping of instructions to comments.
+        """
+        instruction_operand_index: builtins.int
+        """Index into the operand array local to an instruction."""
+        operand_expression_index: builtins.int
+        """Index into the expression array local to an operand, like in Reference.
+        This is not currently used, but allows to implement expression
+        substitutions.
+        """
+        string_table_index: builtins.int
+        """Index into the global string table."""
+        repeatable: builtins.bool
+        """Comment is propagated to all locations that reference the original
+        location.
+        """
+        type: global___BinExport2.Comment.Type.ValueType
+        def __init__(
+            self,
+            *,
+            instruction_index: builtins.int | None = ...,
+            instruction_operand_index: builtins.int | None = ...,
+            operand_expression_index: builtins.int | None = ...,
+            string_table_index: builtins.int | None = ...,
+            repeatable: builtins.bool | None = ...,
+            type: global___BinExport2.Comment.Type.ValueType | None = ...,
+        ) -> None: ...
+        def HasField(self, field_name: typing_extensions.Literal["instruction_index", b"instruction_index", "instruction_operand_index", b"instruction_operand_index", "operand_expression_index", b"operand_expression_index", "repeatable", b"repeatable", "string_table_index", b"string_table_index", "type", b"type"]) -> builtins.bool: ...
+        def ClearField(self, field_name: typing_extensions.Literal["instruction_index", b"instruction_index", "instruction_operand_index", b"instruction_operand_index", "operand_expression_index", b"operand_expression_index", "repeatable", b"repeatable", "string_table_index", b"string_table_index", "type", b"type"]) -> None: ...
+
+    @typing_extensions.final
+    class Section(google.protobuf.message.Message):
+        DESCRIPTOR: google.protobuf.descriptor.Descriptor
+
+        ADDRESS_FIELD_NUMBER: builtins.int
+        SIZE_FIELD_NUMBER: builtins.int
+        FLAG_R_FIELD_NUMBER: builtins.int
+        FLAG_W_FIELD_NUMBER: builtins.int
+        FLAG_X_FIELD_NUMBER: builtins.int
+        address: builtins.int
+        """Section start address."""
+        size: builtins.int
+        """Section size."""
+        flag_r: builtins.bool
+        """Read flag of the section, True when section is readable."""
+        flag_w: builtins.bool
+        """Write flag of the section, True when section is writable."""
+        flag_x: builtins.bool
+        """Execute flag of the section, True when section is executable."""
+        def __init__(
+            self,
+            *,
+            address: builtins.int | None = ...,
+            size: builtins.int | None = ...,
+            flag_r: builtins.bool | None = ...,
+            flag_w: builtins.bool | None = ...,
+            flag_x: builtins.bool | None = ...,
+        ) -> None: ...
+        def HasField(self, field_name: typing_extensions.Literal["address", b"address", "flag_r", b"flag_r", "flag_w", b"flag_w", "flag_x", b"flag_x", "size", b"size"]) -> builtins.bool: ...
+        def ClearField(self, field_name: typing_extensions.Literal["address", b"address", "flag_r", b"flag_r", "flag_w", b"flag_w", "flag_x", b"flag_x", "size", b"size"]) -> None: ...
+
+    @typing_extensions.final
+    class Library(google.protobuf.message.Message):
+        DESCRIPTOR: google.protobuf.descriptor.Descriptor
+
+        IS_STATIC_FIELD_NUMBER: builtins.int
+        LOAD_ADDRESS_FIELD_NUMBER: builtins.int
+        NAME_FIELD_NUMBER: builtins.int
+        is_static: builtins.bool
+        """If this library is statically linked."""
+        load_address: builtins.int
+        """Address where this library was loaded, 0 if unknown."""
+        name: builtins.str
+        """Name of the library (format is platform-dependent)."""
+        def __init__(
+            self,
+            *,
+            is_static: builtins.bool | None = ...,
+            load_address: builtins.int | None = ...,
+            name: builtins.str | None = ...,
+        ) -> None: ...
+        def HasField(self, field_name: typing_extensions.Literal["is_static", b"is_static", "load_address", b"load_address", "name", b"name"]) -> builtins.bool: ...
+        def ClearField(self, field_name: typing_extensions.Literal["is_static", b"is_static", "load_address", b"load_address", "name", b"name"]) -> None: ...
+
+    @typing_extensions.final
+    class Module(google.protobuf.message.Message):
+        DESCRIPTOR: google.protobuf.descriptor.Descriptor
+
+        NAME_FIELD_NUMBER: builtins.int
+        name: builtins.str
+        """Name, such as Java class name. Platform-dependent."""
+        def __init__(
+            self,
+            *,
+            name: builtins.str | None = ...,
+        ) -> None: ...
+        def HasField(self, field_name: typing_extensions.Literal["name", b"name"]) -> builtins.bool: ...
+        def ClearField(self, field_name: typing_extensions.Literal["name", b"name"]) -> None: ...
+
+    META_INFORMATION_FIELD_NUMBER: builtins.int
+    EXPRESSION_FIELD_NUMBER: builtins.int
+    OPERAND_FIELD_NUMBER: builtins.int
+    MNEMONIC_FIELD_NUMBER: builtins.int
+    INSTRUCTION_FIELD_NUMBER: builtins.int
+    BASIC_BLOCK_FIELD_NUMBER: builtins.int
+    FLOW_GRAPH_FIELD_NUMBER: builtins.int
+    CALL_GRAPH_FIELD_NUMBER: builtins.int
+    STRING_TABLE_FIELD_NUMBER: builtins.int
+    ADDRESS_COMMENT_FIELD_NUMBER: builtins.int
+    COMMENT_FIELD_NUMBER: builtins.int
+    STRING_REFERENCE_FIELD_NUMBER: builtins.int
+    EXPRESSION_SUBSTITUTION_FIELD_NUMBER: builtins.int
+    SECTION_FIELD_NUMBER: builtins.int
+    LIBRARY_FIELD_NUMBER: builtins.int
+    DATA_REFERENCE_FIELD_NUMBER: builtins.int
+    MODULE_FIELD_NUMBER: builtins.int
+    @property
+    def meta_information(self) -> global___BinExport2.Meta: ...
+    @property
+    def expression(self) -> google.protobuf.internal.containers.RepeatedCompositeFieldContainer[global___BinExport2.Expression]: ...
+    @property
+    def operand(self) -> google.protobuf.internal.containers.RepeatedCompositeFieldContainer[global___BinExport2.Operand]: ...
+    @property
+    def mnemonic(self) -> google.protobuf.internal.containers.RepeatedCompositeFieldContainer[global___BinExport2.Mnemonic]: ...
+    @property
+    def instruction(self) -> google.protobuf.internal.containers.RepeatedCompositeFieldContainer[global___BinExport2.Instruction]: ...
+    @property
+    def basic_block(self) -> google.protobuf.internal.containers.RepeatedCompositeFieldContainer[global___BinExport2.BasicBlock]: ...
+    @property
+    def flow_graph(self) -> google.protobuf.internal.containers.RepeatedCompositeFieldContainer[global___BinExport2.FlowGraph]: ...
+    @property
+    def call_graph(self) -> global___BinExport2.CallGraph: ...
+    @property
+    def string_table(self) -> google.protobuf.internal.containers.RepeatedScalarFieldContainer[builtins.str]: ...
+    @property
+    def address_comment(self) -> google.protobuf.internal.containers.RepeatedCompositeFieldContainer[global___BinExport2.Reference]:
+        """No longer written. This is here so that BinDiff can work with older
+        BinExport files.
+        """
+    @property
+    def comment(self) -> google.protobuf.internal.containers.RepeatedCompositeFieldContainer[global___BinExport2.Comment]:
+        """Rich comment index used for BinDiff's comment porting."""
+    @property
+    def string_reference(self) -> google.protobuf.internal.containers.RepeatedCompositeFieldContainer[global___BinExport2.Reference]: ...
+    @property
+    def expression_substitution(self) -> google.protobuf.internal.containers.RepeatedCompositeFieldContainer[global___BinExport2.Reference]: ...
+    @property
+    def section(self) -> google.protobuf.internal.containers.RepeatedCompositeFieldContainer[global___BinExport2.Section]: ...
+    @property
+    def library(self) -> google.protobuf.internal.containers.RepeatedCompositeFieldContainer[global___BinExport2.Library]: ...
+    @property
+    def data_reference(self) -> google.protobuf.internal.containers.RepeatedCompositeFieldContainer[global___BinExport2.DataReference]: ...
+    @property
+    def module(self) -> google.protobuf.internal.containers.RepeatedCompositeFieldContainer[global___BinExport2.Module]: ...
+    def __init__(
+        self,
+        *,
+        meta_information: global___BinExport2.Meta | None = ...,
+        expression: collections.abc.Iterable[global___BinExport2.Expression] | None = ...,
+        operand: collections.abc.Iterable[global___BinExport2.Operand] | None = ...,
+        mnemonic: collections.abc.Iterable[global___BinExport2.Mnemonic] | None = ...,
+        instruction: collections.abc.Iterable[global___BinExport2.Instruction] | None = ...,
+        basic_block: collections.abc.Iterable[global___BinExport2.BasicBlock] | None = ...,
+        flow_graph: collections.abc.Iterable[global___BinExport2.FlowGraph] | None = ...,
+        call_graph: global___BinExport2.CallGraph | None = ...,
+        string_table: collections.abc.Iterable[builtins.str] | None = ...,
+        address_comment: collections.abc.Iterable[global___BinExport2.Reference] | None = ...,
+        comment: collections.abc.Iterable[global___BinExport2.Comment] | None = ...,
+        string_reference: collections.abc.Iterable[global___BinExport2.Reference] | None = ...,
+        expression_substitution: collections.abc.Iterable[global___BinExport2.Reference] | None = ...,
+        section: collections.abc.Iterable[global___BinExport2.Section] | None = ...,
+        library: collections.abc.Iterable[global___BinExport2.Library] | None = ...,
+        data_reference: collections.abc.Iterable[global___BinExport2.DataReference] | None = ...,
+        module: collections.abc.Iterable[global___BinExport2.Module] | None = ...,
+    ) -> None: ...
+    def HasField(self, field_name: typing_extensions.Literal["call_graph", b"call_graph", "meta_information", b"meta_information"]) -> builtins.bool: ...
+    def ClearField(self, field_name: typing_extensions.Literal["address_comment", b"address_comment", "basic_block", b"basic_block", "call_graph", b"call_graph", "comment", b"comment", "data_reference", b"data_reference", "expression", b"expression", "expression_substitution", b"expression_substitution", "flow_graph", b"flow_graph", "instruction", b"instruction", "library", b"library", "meta_information", b"meta_information", "mnemonic", b"mnemonic", "module", b"module", "operand", b"operand", "section", b"section", "string_reference", b"string_reference", "string_table", b"string_table"]) -> None: ...
+
+global___BinExport2 = BinExport2

capa/features/extractors/binexport2/extractor.py

@@ -0,0 +1,137 @@
+# Copyright 2023 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import logging
+from typing import Iterator
+
+import capa.features.extractors.elf
+import capa.features.extractors.common
+import capa.features.extractors.binexport2.file
+import capa.features.extractors.binexport2.insn
+import capa.features.extractors.binexport2.helpers
+import capa.features.extractors.binexport2.function
+import capa.features.extractors.binexport2.basicblock
+from capa.features.common import OS, Arch, Format, Feature
+from capa.features.address import Address, AbsoluteVirtualAddress
+from capa.features.extractors.binexport2 import (
+    AddressSpace,
+    AnalysisContext,
+    BinExport2Index,
+    FunctionContext,
+    BasicBlockContext,
+    BinExport2Analysis,
+    InstructionContext,
+)
+from capa.features.extractors.base_extractor import (
+    BBHandle,
+    InsnHandle,
+    SampleHashes,
+    FunctionHandle,
+    StaticFeatureExtractor,
+)
+from capa.features.extractors.binexport2.binexport2_pb2 import BinExport2
+
+logger = logging.getLogger(__name__)
+
+
+class BinExport2FeatureExtractor(StaticFeatureExtractor):
+    def __init__(self, be2: BinExport2, buf: bytes):
+        super().__init__(hashes=SampleHashes.from_bytes(buf))
+        self.be2: BinExport2 = be2
+        self.buf: bytes = buf
+        self.idx: BinExport2Index = BinExport2Index(self.be2)
+        self.analysis: BinExport2Analysis = BinExport2Analysis(self.be2, self.idx, self.buf)
+        address_space: AddressSpace = AddressSpace.from_buf(buf, self.analysis.base_address)
+        self.ctx: AnalysisContext = AnalysisContext(self.buf, self.be2, self.idx, self.analysis, address_space)
+
+        self.global_features: list[tuple[Feature, Address]] = []
+        self.global_features.extend(list(capa.features.extractors.common.extract_format(self.buf)))
+        self.global_features.extend(list(capa.features.extractors.common.extract_os(self.buf)))
+        self.global_features.extend(list(capa.features.extractors.common.extract_arch(self.buf)))
+
+        self.format: set[str] = set()
+        self.os: set[str] = set()
+        self.arch: set[str] = set()
+
+        for feature, _ in self.global_features:
+            assert isinstance(feature.value, str)
+
+            if isinstance(feature, Format):
+                self.format.add(feature.value)
+            elif isinstance(feature, OS):
+                self.os.add(feature.value)
+            elif isinstance(feature, Arch):
+                self.arch.add(feature.value)
+            else:
+                raise ValueError("unexpected global feature: %s", feature)
+
+    def get_base_address(self) -> AbsoluteVirtualAddress:
+        return AbsoluteVirtualAddress(self.analysis.base_address)
+
+    def extract_global_features(self) -> Iterator[tuple[Feature, Address]]:
+        yield from self.global_features
+
+    def extract_file_features(self) -> Iterator[tuple[Feature, Address]]:
+        yield from capa.features.extractors.binexport2.file.extract_features(self.be2, self.buf)
+
+    def get_functions(self) -> Iterator[FunctionHandle]:
+        for flow_graph_index, flow_graph in enumerate(self.be2.flow_graph):
+            entry_basic_block_index: int = flow_graph.entry_basic_block_index
+            flow_graph_address: int = self.idx.get_basic_block_address(entry_basic_block_index)
+
+            vertex_idx: int = self.idx.vertex_index_by_address[flow_graph_address]
+            be2_vertex: BinExport2.CallGraph.Vertex = self.be2.call_graph.vertex[vertex_idx]
+
+            # skip thunks
+            if capa.features.extractors.binexport2.helpers.is_vertex_type(
+                be2_vertex, BinExport2.CallGraph.Vertex.Type.THUNK
+            ):
+                continue
+
+            yield FunctionHandle(
+                AbsoluteVirtualAddress(flow_graph_address),
+                inner=FunctionContext(self.ctx, flow_graph_index, self.format, self.os, self.arch),
+            )
+
+    def extract_function_features(self, fh: FunctionHandle) -> Iterator[tuple[Feature, Address]]:
+        yield from capa.features.extractors.binexport2.function.extract_features(fh)
+
+    def get_basic_blocks(self, fh: FunctionHandle) -> Iterator[BBHandle]:
+        fhi: FunctionContext = fh.inner
+        flow_graph_index: int = fhi.flow_graph_index
+        flow_graph: BinExport2.FlowGraph = self.be2.flow_graph[flow_graph_index]
+
+        for basic_block_index in flow_graph.basic_block_index:
+            basic_block_address: int = self.idx.get_basic_block_address(basic_block_index)
+            yield BBHandle(
+                address=AbsoluteVirtualAddress(basic_block_address),
+                inner=BasicBlockContext(basic_block_index),
+            )
+
+    def extract_basic_block_features(self, fh: FunctionHandle, bbh: BBHandle) -> Iterator[tuple[Feature, Address]]:
+        yield from capa.features.extractors.binexport2.basicblock.extract_features(fh, bbh)
+
+    def get_instructions(self, fh: FunctionHandle, bbh: BBHandle) -> Iterator[InsnHandle]:
+        bbi: BasicBlockContext = bbh.inner
+        basic_block: BinExport2.BasicBlock = self.be2.basic_block[bbi.basic_block_index]
+        for instruction_index, _, instruction_address in self.idx.basic_block_instructions(basic_block):
+            yield InsnHandle(
+                address=AbsoluteVirtualAddress(instruction_address),
+                inner=InstructionContext(instruction_index),
+            )
+
+    def extract_insn_features(
+        self, fh: FunctionHandle, bbh: BBHandle, ih: InsnHandle
+    ) -> Iterator[tuple[Feature, Address]]:
+        yield from capa.features.extractors.binexport2.insn.extract_features(fh, bbh, ih)

capa/features/extractors/binexport2/file.py

@@ -0,0 +1,87 @@
+# Copyright 2023 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import io
+import logging
+from typing import Iterator
+
+import pefile
+from elftools.elf.elffile import ELFFile
+
+import capa.features.common
+import capa.features.extractors.common
+import capa.features.extractors.pefile
+import capa.features.extractors.elffile
+from capa.features.common import Feature
+from capa.features.address import Address
+from capa.features.extractors.binexport2.binexport2_pb2 import BinExport2
+
+logger = logging.getLogger(__name__)
+
+
+def extract_file_export_names(_be2: BinExport2, buf: bytes) -> Iterator[tuple[Feature, Address]]:
+    if buf.startswith(capa.features.extractors.common.MATCH_PE):
+        pe: pefile.PE = pefile.PE(data=buf)
+        yield from capa.features.extractors.pefile.extract_file_export_names(pe)
+    elif buf.startswith(capa.features.extractors.common.MATCH_ELF):
+        elf: ELFFile = ELFFile(io.BytesIO(buf))
+        yield from capa.features.extractors.elffile.extract_file_export_names(elf)
+    else:
+        logger.warning("unsupported format")
+
+
+def extract_file_import_names(_be2: BinExport2, buf: bytes) -> Iterator[tuple[Feature, Address]]:
+    if buf.startswith(capa.features.extractors.common.MATCH_PE):
+        pe: pefile.PE = pefile.PE(data=buf)
+        yield from capa.features.extractors.pefile.extract_file_import_names(pe)
+    elif buf.startswith(capa.features.extractors.common.MATCH_ELF):
+        elf: ELFFile = ELFFile(io.BytesIO(buf))
+        yield from capa.features.extractors.elffile.extract_file_import_names(elf)
+    else:
+        logger.warning("unsupported format")
+
+
+def extract_file_section_names(_be2: BinExport2, buf: bytes) -> Iterator[tuple[Feature, Address]]:
+    if buf.startswith(capa.features.extractors.common.MATCH_PE):
+        pe: pefile.PE = pefile.PE(data=buf)
+        yield from capa.features.extractors.pefile.extract_file_section_names(pe)
+    elif buf.startswith(capa.features.extractors.common.MATCH_ELF):
+        elf: ELFFile = ELFFile(io.BytesIO(buf))
+        yield from capa.features.extractors.elffile.extract_file_section_names(elf)
+    else:
+        logger.warning("unsupported format")
+
+
+def extract_file_strings(_be2: BinExport2, buf: bytes) -> Iterator[tuple[Feature, Address]]:
+    yield from capa.features.extractors.common.extract_file_strings(buf)
+
+
+def extract_file_format(_be2: BinExport2, buf: bytes) -> Iterator[tuple[Feature, Address]]:
+    yield from capa.features.extractors.common.extract_format(buf)
+
+
+def extract_features(be2: BinExport2, buf: bytes) -> Iterator[tuple[Feature, Address]]:
+    """extract file features"""
+    for file_handler in FILE_HANDLERS:
+        for feature, addr in file_handler(be2, buf):
+            yield feature, addr
+
+
+FILE_HANDLERS = (
+    extract_file_export_names,
+    extract_file_import_names,
+    extract_file_strings,
+    extract_file_section_names,
+    extract_file_format,
+)

capa/features/extractors/binexport2/function.py

@@ -0,0 +1,79 @@
+# Copyright 2023 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from typing import Iterator
+
+from capa.features.file import FunctionName
+from capa.features.common import Feature, Characteristic
+from capa.features.address import Address, AbsoluteVirtualAddress
+from capa.features.extractors import loops
+from capa.features.extractors.binexport2 import BinExport2Index, FunctionContext
+from capa.features.extractors.base_extractor import FunctionHandle
+from capa.features.extractors.binexport2.binexport2_pb2 import BinExport2
+
+
+def extract_function_calls_to(fh: FunctionHandle) -> Iterator[tuple[Feature, Address]]:
+    fhi: FunctionContext = fh.inner
+
+    be2: BinExport2 = fhi.ctx.be2
+    idx: BinExport2Index = fhi.ctx.idx
+
+    flow_graph_index: int = fhi.flow_graph_index
+    flow_graph_address: int = idx.flow_graph_address_by_index[flow_graph_index]
+    vertex_index: int = idx.vertex_index_by_address[flow_graph_address]
+
+    for caller_index in idx.callers_by_vertex_index[vertex_index]:
+        caller: BinExport2.CallGraph.Vertex = be2.call_graph.vertex[caller_index]
+        caller_address: int = caller.address
+        yield Characteristic("calls to"), AbsoluteVirtualAddress(caller_address)
+
+
+def extract_function_loop(fh: FunctionHandle) -> Iterator[tuple[Feature, Address]]:
+    fhi: FunctionContext = fh.inner
+
+    be2: BinExport2 = fhi.ctx.be2
+
+    flow_graph_index: int = fhi.flow_graph_index
+    flow_graph: BinExport2.FlowGraph = be2.flow_graph[flow_graph_index]
+
+    edges: list[tuple[int, int]] = []
+    for edge in flow_graph.edge:
+        edges.append((edge.source_basic_block_index, edge.target_basic_block_index))
+
+    if loops.has_loop(edges):
+        yield Characteristic("loop"), fh.address
+
+
+def extract_function_name(fh: FunctionHandle) -> Iterator[tuple[Feature, Address]]:
+    fhi: FunctionContext = fh.inner
+
+    be2: BinExport2 = fhi.ctx.be2
+    idx: BinExport2Index = fhi.ctx.idx
+    flow_graph_index: int = fhi.flow_graph_index
+
+    flow_graph_address: int = idx.flow_graph_address_by_index[flow_graph_index]
+    vertex_index: int = idx.vertex_index_by_address[flow_graph_address]
+    vertex: BinExport2.CallGraph.Vertex = be2.call_graph.vertex[vertex_index]
+
+    if vertex.HasField("mangled_name"):
+        yield FunctionName(vertex.mangled_name), fh.address
+
+
+def extract_features(fh: FunctionHandle) -> Iterator[tuple[Feature, Address]]:
+    for func_handler in FUNCTION_HANDLERS:
+        for feature, addr in func_handler(fh):
+            yield feature, addr
+
+
+FUNCTION_HANDLERS = (extract_function_calls_to, extract_function_loop, extract_function_name)

capa/features/extractors/binexport2/helpers.py

@@ -0,0 +1,699 @@
+# Copyright 2024 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import re
+from typing import Union, Iterator, Optional
+from collections import defaultdict
+from dataclasses import dataclass
+
+import capa.features.extractors.helpers
+import capa.features.extractors.binexport2.helpers
+from capa.features.common import ARCH_I386, ARCH_AMD64, ARCH_AARCH64
+from capa.features.extractors.binexport2.binexport2_pb2 import BinExport2
+
+HAS_ARCH32 = {ARCH_I386}
+HAS_ARCH64 = {ARCH_AARCH64, ARCH_AMD64}
+
+HAS_ARCH_INTEL = {ARCH_I386, ARCH_AMD64}
+HAS_ARCH_ARM = {ARCH_AARCH64}
+
+
+def mask_immediate(arch: set[str], immediate: int) -> int:
+    if arch & HAS_ARCH64:
+        immediate &= 0xFFFFFFFFFFFFFFFF
+    elif arch & HAS_ARCH32:
+        immediate &= 0xFFFFFFFF
+    return immediate
+
+
+def twos_complement(arch: set[str], immediate: int, default: Optional[int] = None) -> int:
+    if default is not None:
+        return capa.features.extractors.helpers.twos_complement(immediate, default)
+    elif arch & HAS_ARCH64:
+        return capa.features.extractors.helpers.twos_complement(immediate, 64)
+    elif arch & HAS_ARCH32:
+        return capa.features.extractors.helpers.twos_complement(immediate, 32)
+    return immediate
+
+
+def is_address_mapped(be2: BinExport2, address: int) -> bool:
+    """return True if the given address is mapped"""
+    sections_with_perms: Iterator[BinExport2.Section] = filter(lambda s: s.flag_r or s.flag_w or s.flag_x, be2.section)
+    return any(section.address <= address < section.address + section.size for section in sections_with_perms)
+
+
+def is_vertex_type(vertex: BinExport2.CallGraph.Vertex, type_: BinExport2.CallGraph.Vertex.Type.ValueType) -> bool:
+    return vertex.HasField("type") and vertex.type == type_
+
+
+# internal to `build_expression_tree`
+# this is unstable: it is subject to change, so don't rely on it!
+def _prune_expression_tree_references_to_tree_index(
+    expression_tree: list[list[int]],
+    tree_index: int,
+):
+    # `i` is the index of the tree node that we'll search for `tree_index`
+    # if we remove `tree_index` from it, and it is now empty,
+    # then we'll need to prune references to `i`.
+    for i, tree_node in enumerate(expression_tree):
+        if tree_index in tree_node:
+            tree_node.remove(tree_index)
+
+            if len(tree_node) == 0:
+                # if the parent node is now empty,
+                # remove references to that parent node.
+                _prune_expression_tree_references_to_tree_index(expression_tree, i)
+
+
+# internal to `build_expression_tree`
+# this is unstable: it is subject to change, so don't rely on it!
+def _prune_expression_tree_empty_shifts(
+    be2: BinExport2,
+    operand: BinExport2.Operand,
+    expression_tree: list[list[int]],
+    tree_index: int,
+):
+    expression_index = operand.expression_index[tree_index]
+    expression = be2.expression[expression_index]
+    children_tree_indexes: list[int] = expression_tree[tree_index]
+
+    if expression.type == BinExport2.Expression.OPERATOR:
+        if len(children_tree_indexes) == 0 and expression.symbol in ("lsl", "lsr"):
+            # Ghidra may emit superfluous lsl nodes with no children.
+            # https://github.com/mandiant/capa/pull/2340/files#r1750003919
+            # Which is maybe: https://github.com/NationalSecurityAgency/ghidra/issues/6821#issuecomment-2295394697
+            #
+            # Which seems to be as if the shift wasn't there (shift of #0)
+            # so we want to remove references to this node from any parent nodes.
+            _prune_expression_tree_references_to_tree_index(expression_tree, tree_index)
+
+            return
+
+    for child_tree_index in children_tree_indexes:
+        _prune_expression_tree_empty_shifts(be2, operand, expression_tree, child_tree_index)
+
+
+# internal to `build_expression_tree`
+# this is unstable: it is subject to change, so don't rely on it!
+def _fixup_expression_tree_references_to_tree_index(
+    expression_tree: list[list[int]],
+    existing_index: int,
+    new_index: int,
+):
+    for tree_node in expression_tree:
+        for i, index in enumerate(tree_node):
+            if index == existing_index:
+                tree_node[i] = new_index
+
+
+# internal to `build_expression_tree`
+# this is unstable: it is subject to change, so don't rely on it!
+def _fixup_expression_tree_lonely_commas(
+    be2: BinExport2,
+    operand: BinExport2.Operand,
+    expression_tree: list[list[int]],
+    tree_index: int,
+):
+    expression_index = operand.expression_index[tree_index]
+    expression = be2.expression[expression_index]
+    children_tree_indexes: list[int] = expression_tree[tree_index]
+
+    if expression.type == BinExport2.Expression.OPERATOR:
+        if len(children_tree_indexes) == 1 and expression.symbol == ",":
+            existing_index = tree_index
+            new_index = children_tree_indexes[0]
+            _fixup_expression_tree_references_to_tree_index(expression_tree, existing_index, new_index)
+
+    for child_tree_index in children_tree_indexes:
+        _fixup_expression_tree_lonely_commas(be2, operand, expression_tree, child_tree_index)
+
+
+# internal to `build_expression_tree`
+# this is unstable: it is subject to change, so don't rely on it!
+def _prune_expression_tree(
+    be2: BinExport2,
+    operand: BinExport2.Operand,
+    expression_tree: list[list[int]],
+):
+    _prune_expression_tree_empty_shifts(be2, operand, expression_tree, 0)
+    _fixup_expression_tree_lonely_commas(be2, operand, expression_tree, 0)
+
+
+# this is unstable: it is subject to change, so don't rely on it!
+def _build_expression_tree(
+    be2: BinExport2,
+    operand: BinExport2.Operand,
+) -> list[list[int]]:
+    # The reconstructed expression tree layout, linking parent nodes to their children.
+    #
+    # There is one list of integers for each expression in the operand.
+    # These integers are indexes of other expressions in the same operand,
+    # which are the children of that expression.
+    #
+    # So:
+    #
+    #   [ [1, 3], [2], [], [4], [5], []]
+    #
+    # means the first expression has two children, at index 1 and 3,
+    # and the tree looks like:
+    #
+    #        0
+    #       / \
+    #      1   3
+    #      |   |
+    #      2   4
+    #          |
+    #          5
+    #
+    # Remember, these are the indices into the entries in operand.expression_index.
+    if len(operand.expression_index) == 0:
+        # Ghidra bug where empty operands (no expressions) may
+        # exist (see https://github.com/NationalSecurityAgency/ghidra/issues/6817)
+        return []
+
+    tree: list[list[int]] = []
+    for i, expression_index in enumerate(operand.expression_index):
+        children = []
+
+        # scan all subsequent expressions, looking for those that have parent_index == current.expression_index
+        for j, candidate_index in enumerate(operand.expression_index[i + 1 :]):
+            candidate = be2.expression[candidate_index]
+
+            if candidate.parent_index == expression_index:
+                children.append(i + j + 1)
+
+        tree.append(children)
+
+    _prune_expression_tree(be2, operand, tree)
+
+    return tree
+
+
+def _fill_operand_expression_list(
+    be2: BinExport2,
+    operand: BinExport2.Operand,
+    expression_tree: list[list[int]],
+    tree_index: int,
+    expression_list: list[BinExport2.Expression],
+):
+    """
+    Walk the given expression tree and collect the expression nodes in-order.
+    """
+    expression_index = operand.expression_index[tree_index]
+    expression = be2.expression[expression_index]
+    children_tree_indexes: list[int] = expression_tree[tree_index]
+
+    if expression.type == BinExport2.Expression.REGISTER:
+        assert len(children_tree_indexes) <= 1
+        expression_list.append(expression)
+
+        if len(children_tree_indexes) == 0:
+            return
+        elif len(children_tree_indexes) == 1:
+            # like for aarch64 with vector instructions, indicating vector data size:
+            #
+            #     FADD V0.4S, V1.4S, V2.4S
+            #
+            # see: https://github.com/mandiant/capa/issues/2528
+            child_index = children_tree_indexes[0]
+            _fill_operand_expression_list(be2, operand, expression_tree, child_index, expression_list)
+            return
+        else:
+            raise NotImplementedError(len(children_tree_indexes))
+
+    elif expression.type == BinExport2.Expression.SYMBOL:
+        assert len(children_tree_indexes) <= 1
+        expression_list.append(expression)
+
+        if len(children_tree_indexes) == 0:
+            return
+        elif len(children_tree_indexes) == 1:
+            # like: v
+            # from: mov v0.D[0x1], x9
+            #           |
+            #           0
+            #           .
+            #           |
+            #           D
+            child_index = children_tree_indexes[0]
+            _fill_operand_expression_list(be2, operand, expression_tree, child_index, expression_list)
+            return
+        else:
+            raise NotImplementedError(len(children_tree_indexes))
+
+    elif expression.type == BinExport2.Expression.IMMEDIATE_INT:
+        assert len(children_tree_indexes) <= 1
+        expression_list.append(expression)
+
+        if len(children_tree_indexes) == 0:
+            return
+        elif len(children_tree_indexes) == 1:
+            # the ghidra exporter can produce some weird expressions,
+            # particularly for MSRs, like for:
+            #
+            #     sreg(3, 0, c.0, c.4, 4)
+            #
+            # see: https://github.com/mandiant/capa/issues/2530
+            child_index = children_tree_indexes[0]
+            _fill_operand_expression_list(be2, operand, expression_tree, child_index, expression_list)
+            return
+        else:
+            raise NotImplementedError(len(children_tree_indexes))
+
+    elif expression.type == BinExport2.Expression.SIZE_PREFIX:
+        # like: b4
+        #
+        # We might want to use this occasionally, such as to disambiguate the
+        # size of MOVs into/out of memory. But I'm not sure when/where we need that yet.
+        #
+        # IDA spams this size prefix hint *everywhere*, so we can't rely on the exporter
+        # to provide it only when necessary.
+        assert len(children_tree_indexes) == 1
+        child_index = children_tree_indexes[0]
+        _fill_operand_expression_list(be2, operand, expression_tree, child_index, expression_list)
+        return
+
+    elif expression.type == BinExport2.Expression.OPERATOR:
+        if len(children_tree_indexes) == 1:
+            # prefix operator, like "ds:"
+            expression_list.append(expression)
+            child_index = children_tree_indexes[0]
+            _fill_operand_expression_list(be2, operand, expression_tree, child_index, expression_list)
+            return
+
+        elif len(children_tree_indexes) == 2:
+            # infix operator: like "+" in "ebp+10"
+            child_a = children_tree_indexes[0]
+            child_b = children_tree_indexes[1]
+            _fill_operand_expression_list(be2, operand, expression_tree, child_a, expression_list)
+            expression_list.append(expression)
+            _fill_operand_expression_list(be2, operand, expression_tree, child_b, expression_list)
+            return
+
+        elif len(children_tree_indexes) == 3:
+            # infix operator: like "+" in "ebp+ecx+10"
+            child_a = children_tree_indexes[0]
+            child_b = children_tree_indexes[1]
+            child_c = children_tree_indexes[2]
+            _fill_operand_expression_list(be2, operand, expression_tree, child_a, expression_list)
+            expression_list.append(expression)
+            _fill_operand_expression_list(be2, operand, expression_tree, child_b, expression_list)
+            expression_list.append(expression)
+            _fill_operand_expression_list(be2, operand, expression_tree, child_c, expression_list)
+            return
+
+        else:
+            raise NotImplementedError(len(children_tree_indexes))
+
+    elif expression.type == BinExport2.Expression.DEREFERENCE:
+        assert len(children_tree_indexes) == 1
+        expression_list.append(expression)
+
+        child_index = children_tree_indexes[0]
+        _fill_operand_expression_list(be2, operand, expression_tree, child_index, expression_list)
+        return
+
+    elif expression.type == BinExport2.Expression.IMMEDIATE_FLOAT:
+        raise NotImplementedError(expression.type)
+
+    else:
+        raise NotImplementedError(expression.type)
+
+
+def get_operand_expressions(be2: BinExport2, op: BinExport2.Operand) -> list[BinExport2.Expression]:
+    tree = _build_expression_tree(be2, op)
+
+    expressions: list[BinExport2.Expression] = []
+    _fill_operand_expression_list(be2, op, tree, 0, expressions)
+
+    return expressions
+
+
+def get_operand_register_expression(be2: BinExport2, operand: BinExport2.Operand) -> Optional[BinExport2.Expression]:
+    if len(operand.expression_index) == 1:
+        expression: BinExport2.Expression = be2.expression[operand.expression_index[0]]
+        if expression.type == BinExport2.Expression.REGISTER:
+            return expression
+    return None
+
+
+def get_operand_immediate_expression(be2: BinExport2, operand: BinExport2.Operand) -> Optional[BinExport2.Expression]:
+    if len(operand.expression_index) == 1:
+        # - type: IMMEDIATE_INT
+        #   immediate: 20588728364
+        #   parent_index: 0
+        expression: BinExport2.Expression = be2.expression[operand.expression_index[0]]
+        if expression.type == BinExport2.Expression.IMMEDIATE_INT:
+            return expression
+
+    elif len(operand.expression_index) == 2:
+        # from IDA, which provides a size hint for every operand,
+        # we get the following pattern for immediate constants:
+        #
+        # - type: SIZE_PREFIX
+        #   symbol: "b8"
+        # - type: IMMEDIATE_INT
+        #   immediate: 20588728364
+        #   parent_index: 0
+        expression0: BinExport2.Expression = be2.expression[operand.expression_index[0]]
+        expression1: BinExport2.Expression = be2.expression[operand.expression_index[1]]
+
+        if expression0.type == BinExport2.Expression.SIZE_PREFIX:
+            if expression1.type == BinExport2.Expression.IMMEDIATE_INT:
+                return expression1
+
+    return None
+
+
+def get_instruction_mnemonic(be2: BinExport2, instruction: BinExport2.Instruction) -> str:
+    return be2.mnemonic[instruction.mnemonic_index].name.lower()
+
+
+def get_instruction_operands(be2: BinExport2, instruction: BinExport2.Instruction) -> list[BinExport2.Operand]:
+    return [be2.operand[operand_index] for operand_index in instruction.operand_index]
+
+
+def split_with_delimiters(s: str, delimiters: tuple[str, ...]) -> Iterator[str]:
+    """
+    Splits a string by any of the provided delimiter characters,
+    including the delimiters in the results.
+
+    Args:
+        string: The string to split.
+        delimiters: A string containing the characters to use as delimiters.
+    """
+    start = 0
+    for i, char in enumerate(s):
+        if char in delimiters:
+            yield s[start:i]
+            yield char
+            start = i + 1
+
+    if start < len(s):
+        yield s[start:]
+
+
+BinExport2OperandPattern = Union[str, tuple[str, ...]]
+
+
+@dataclass
+class BinExport2InstructionPattern:
+    """
+    This describes a way to match disassembled instructions, with mnemonics and operands.
+
+    You can specify constraints on the instruction, via:
+      - the mnemonics, like "mov",
+      - number of operands, and
+      - format of each operand, "[reg, reg, #int]".
+
+    During matching, you can also capture a single element, to see its concrete value.
+    For example, given the pattern:
+
+        mov reg0, #int0  ; capture int0
+
+    and the instruction:
+
+        mov eax, 1
+
+    Then the capture will contain the immediate integer 1.
+
+    This matcher uses the BinExport2 data layout under the hood.
+    """
+
+    mnemonics: tuple[str, ...]
+    operands: tuple[Union[str, BinExport2OperandPattern], ...]
+    capture: Optional[str]
+
+    @classmethod
+    def from_str(cls, query: str):
+        """
+        Parse a pattern string into a Pattern instance.
+        The supported syntax is like this:
+
+            br      reg
+            br      reg                          ; capture reg
+            br      reg(stack)                   ; capture reg
+            br      reg(not-stack)               ; capture reg
+            mov     reg0, reg1                   ; capture reg0
+            adrp    reg, #int                    ; capture #int
+            add     reg, reg, #int               ; capture #int
+            ldr     reg0, [reg1]                 ; capture reg1
+            ldr|str reg, [reg, #int]             ; capture #int
+            ldr|str reg, [reg(stack), #int]      ; capture #int
+            ldr|str reg, [reg(not-stack), #int]  ; capture #int
+            ldr|str reg, [reg, #int]!            ; capture #int
+            ldr|str reg, [reg], #int             ; capture #int
+            ldp|stp reg, reg, [reg, #int]        ; capture #int
+            ldp|stp reg, reg, [reg, #int]!       ; capture #int
+            ldp|stp reg, reg, [reg], #int        ; capture #int
+        """
+        #
+        # The implementation of the parser here is obviously ugly.
+        # Its handwritten and probably fragile. But since we don't
+        # expect this to be widely used, its probably ok.
+        # Don't hesitate to rewrite this if it becomes more important.
+        #
+        # Note that this doesn't have to be very performant.
+        # We expect these patterns to be parsed once upfront and then reused
+        # (globally at the module level?) rather than within any loop.
+        #
+
+        pattern, _, comment = query.strip().partition(";")
+
+        # we don't support fs: yet
+        assert ":" not in pattern
+
+        # from "capture #int" to "#int"
+        if comment:
+            comment = comment.strip()
+            assert comment.startswith("capture ")
+            capture = comment[len("capture ") :]
+        else:
+            capture = None
+
+        # from "ldr|str ..." to ["ldr", "str"]
+        pattern = pattern.strip()
+        mnemonic, _, rest = pattern.partition(" ")
+        mnemonics = mnemonic.split("|")
+
+        operands: list[Union[str, tuple[str, ...]]] = []
+        while rest:
+            rest = rest.strip()
+            if not rest.startswith("["):
+                # If its not a dereference, which looks like `[op, op, op, ...]`,
+                # then its a simple operand, which we can split by the next comma.
+                operand, _, rest = rest.partition(", ")
+                rest = rest.strip()
+                operands.append(operand)
+
+            else:
+                # This looks like a dereference, something like `[op, op, op, ...]`.
+                # Since these can't be nested, look for the next ] and then parse backwards.
+                deref_end = rest.index("]")
+                try:
+                    deref_end = rest.index(", ", deref_end)
+                    deref_end += len(", ")
+                except ValueError:
+                    deref = rest
+                    rest = ""
+                else:
+                    deref = rest[:deref_end]
+                    rest = rest[deref_end:]
+                    rest = rest.strip()
+                    deref = deref.rstrip(" ")
+                    deref = deref.rstrip(",")
+
+                # like: [reg, #int]!
+                has_postindex_writeback = deref.endswith("!")
+
+                deref = deref.rstrip("!")
+                deref = deref.rstrip("]")
+                deref = deref.lstrip("[")
+
+                parts = tuple(split_with_delimiters(deref, (",", "+", "*")))
+                parts = tuple(s.strip() for s in parts)
+
+                # emit operands in this order to match
+                # how BinExport2 expressions are flatted
+                # by get_operand_expressions
+                if has_postindex_writeback:
+                    operands.append(("!", "[") + parts)
+                else:
+                    operands.append(("[",) + parts)
+
+        for operand in operands:  # type: ignore
+            # Try to ensure we've parsed the operands correctly.
+            # This is just sanity checking.
+            for o in (operand,) if isinstance(operand, str) else operand:
+                # operands can look like:
+                #  - reg
+                #  - reg0
+                #  - reg(stack)
+                #  - reg0(stack)
+                #  - reg(not-stack)
+                #  - reg0(not-stack)
+                #  - #int
+                #  - #int0
+                # and a limited set of supported operators.
+                # use an inline regex so that its easy to read. not perf critical.
+                assert re.match(r"^(reg|#int)[0-9]?(\(stack\)|\(not-stack\))?$", o) or o in ("[", ",", "!", "+", "*")
+
+        return cls(tuple(mnemonics), tuple(operands), capture)
+
+    @dataclass
+    class MatchResult:
+        operand_index: int
+        expression_index: int
+        expression: BinExport2.Expression
+
+    def match(
+        self, mnemonic: str, operand_expressions: list[list[BinExport2.Expression]]
+    ) -> Optional["BinExport2InstructionPattern.MatchResult"]:
+        """
+        Match the given BinExport2 data against this pattern.
+
+        The BinExport2 expression tree must have been flattened, such as with
+        capa.features.extractors.binexport2.helpers.get_operand_expressions.
+
+        If there's a match, the captured Expression instance is returned.
+        Otherwise, you get None back.
+        """
+        if mnemonic not in self.mnemonics:
+            return None
+
+        if len(self.operands) != len(operand_expressions):
+            return None
+
+        captured = None
+
+        for operand_index, found_expressions in enumerate(operand_expressions):
+            wanted_expressions = self.operands[operand_index]
+
+            # from `"reg"` to `("reg", )`
+            if isinstance(wanted_expressions, str):
+                wanted_expressions = (wanted_expressions,)
+            assert isinstance(wanted_expressions, tuple)
+
+            if len(wanted_expressions) != len(found_expressions):
+                return None
+
+            for expression_index, (wanted_expression, found_expression) in enumerate(
+                zip(wanted_expressions, found_expressions)
+            ):
+                if wanted_expression.startswith("reg"):
+                    if found_expression.type != BinExport2.Expression.REGISTER:
+                        return None
+
+                    if wanted_expression.endswith(")"):
+                        if wanted_expression.endswith("(not-stack)"):
+                            # intel 64: rsp, esp, sp,
+                            # intel 32: ebp, ebp, bp
+                            # arm: sp
+                            register_name = found_expression.symbol.lower()
+                            if register_name in ("rsp", "esp", "sp", "rbp", "ebp", "bp"):
+                                return None
+
+                        elif wanted_expression.endswith("(stack)"):
+                            register_name = found_expression.symbol.lower()
+                            if register_name not in ("rsp", "esp", "sp", "rbp", "ebp", "bp"):
+                                return None
+
+                        else:
+                            raise ValueError("unexpected expression suffix", wanted_expression)
+
+                    if self.capture == wanted_expression:
+                        captured = BinExport2InstructionPattern.MatchResult(
+                            operand_index, expression_index, found_expression
+                        )
+
+                elif wanted_expression.startswith("#int"):
+                    if found_expression.type != BinExport2.Expression.IMMEDIATE_INT:
+                        return None
+
+                    if self.capture == wanted_expression:
+                        captured = BinExport2InstructionPattern.MatchResult(
+                            operand_index, expression_index, found_expression
+                        )
+
+                elif wanted_expression == "[":
+                    if found_expression.type != BinExport2.Expression.DEREFERENCE:
+                        return None
+
+                elif wanted_expression in (",", "!", "+", "*"):
+                    if found_expression.type != BinExport2.Expression.OPERATOR:
+                        return None
+
+                    if found_expression.symbol != wanted_expression:
+                        return None
+
+                else:
+                    raise ValueError(found_expression)
+
+        if captured:
+            return captured
+        else:
+            # There were no captures, so
+            # return arbitrary non-None expression
+            return BinExport2InstructionPattern.MatchResult(operand_index, expression_index, found_expression)
+
+
+class BinExport2InstructionPatternMatcher:
+    """Index and match a collection of instruction patterns."""
+
+    def __init__(self, queries: list[BinExport2InstructionPattern]):
+        self.queries = queries
+        # shard the patterns by (mnemonic, #operands)
+        self._index: dict[tuple[str, int], list[BinExport2InstructionPattern]] = defaultdict(list)
+
+        for query in queries:
+            for mnemonic in query.mnemonics:
+                self._index[(mnemonic.lower(), len(query.operands))].append(query)
+
+    @classmethod
+    def from_str(cls, patterns: str):
+        return cls(
+            [
+                BinExport2InstructionPattern.from_str(line)
+                for line in filter(
+                    lambda line: not line.startswith("#"), (line.strip() for line in patterns.split("\n"))
+                )
+            ]
+        )
+
+    def match(
+        self, mnemonic: str, operand_expressions: list[list[BinExport2.Expression]]
+    ) -> Optional[BinExport2InstructionPattern.MatchResult]:
+        queries = self._index.get((mnemonic.lower(), len(operand_expressions)), [])
+        for query in queries:
+            captured = query.match(mnemonic.lower(), operand_expressions)
+            if captured:
+                return captured
+
+        return None
+
+    def match_with_be2(
+        self, be2: BinExport2, instruction_index: int
+    ) -> Optional[BinExport2InstructionPattern.MatchResult]:
+        instruction: BinExport2.Instruction = be2.instruction[instruction_index]
+        mnemonic: str = get_instruction_mnemonic(be2, instruction)
+
+        if (mnemonic.lower(), len(instruction.operand_index)) not in self._index:
+            # verify that we might have a hit before we realize the operand expression list
+            return None
+
+        operands = []
+        for operand_index in instruction.operand_index:
+            operands.append(get_operand_expressions(be2, be2.operand[operand_index]))
+
+        return self.match(mnemonic, operands)

capa/features/extractors/binexport2/insn.py

@@ -0,0 +1,261 @@
+# Copyright 2023 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import logging
+from typing import Iterator
+
+import capa.features.extractors.helpers
+import capa.features.extractors.strings
+import capa.features.extractors.binexport2.helpers
+import capa.features.extractors.binexport2.arch.arm.insn
+import capa.features.extractors.binexport2.arch.intel.insn
+from capa.features.insn import API, Mnemonic
+from capa.features.common import Bytes, String, Feature, Characteristic
+from capa.features.address import Address, AbsoluteVirtualAddress
+from capa.features.extractors.binexport2 import (
+    AddressSpace,
+    AnalysisContext,
+    BinExport2Index,
+    FunctionContext,
+    ReadMemoryError,
+    BinExport2Analysis,
+    InstructionContext,
+)
+from capa.features.extractors.base_extractor import BBHandle, InsnHandle, FunctionHandle
+from capa.features.extractors.binexport2.helpers import HAS_ARCH_ARM, HAS_ARCH_INTEL
+from capa.features.extractors.binexport2.binexport2_pb2 import BinExport2
+
+logger = logging.getLogger(__name__)
+
+
+def extract_insn_api_features(fh: FunctionHandle, _bbh: BBHandle, ih: InsnHandle) -> Iterator[tuple[Feature, Address]]:
+    fhi: FunctionContext = fh.inner
+    ii: InstructionContext = ih.inner
+
+    be2: BinExport2 = fhi.ctx.be2
+    be2_index: BinExport2Index = fhi.ctx.idx
+    be2_analysis: BinExport2Analysis = fhi.ctx.analysis
+    insn: BinExport2.Instruction = be2.instruction[ii.instruction_index]
+
+    for addr in insn.call_target:
+        addr = be2_analysis.thunks.get(addr, addr)
+
+        if addr not in be2_index.vertex_index_by_address:
+            # disassembler did not define function at address
+            logger.debug("0x%x is not a vertex", addr)
+            continue
+
+        vertex_idx: int = be2_index.vertex_index_by_address[addr]
+        vertex: BinExport2.CallGraph.Vertex = be2.call_graph.vertex[vertex_idx]
+
+        if not capa.features.extractors.binexport2.helpers.is_vertex_type(
+            vertex, BinExport2.CallGraph.Vertex.Type.IMPORTED
+        ):
+            continue
+
+        if not vertex.HasField("mangled_name"):
+            logger.debug("vertex %d does not have mangled_name", vertex_idx)
+            continue
+
+        api_name: str = vertex.mangled_name
+        for name in capa.features.extractors.helpers.generate_symbols("", api_name):
+            yield API(name), ih.address
+
+
+def extract_insn_number_features(
+    fh: FunctionHandle, bbh: BBHandle, ih: InsnHandle
+) -> Iterator[tuple[Feature, Address]]:
+    fhi: FunctionContext = fh.inner
+
+    if fhi.arch & HAS_ARCH_INTEL:
+        yield from capa.features.extractors.binexport2.arch.intel.insn.extract_insn_number_features(fh, bbh, ih)
+    elif fhi.arch & HAS_ARCH_ARM:
+        yield from capa.features.extractors.binexport2.arch.arm.insn.extract_insn_number_features(fh, bbh, ih)
+
+
+def extract_insn_bytes_features(fh: FunctionHandle, bbh: BBHandle, ih: InsnHandle) -> Iterator[tuple[Feature, Address]]:
+    fhi: FunctionContext = fh.inner
+    ii: InstructionContext = ih.inner
+
+    ctx: AnalysisContext = fhi.ctx
+    be2: BinExport2 = ctx.be2
+    idx: BinExport2Index = ctx.idx
+    address_space: AddressSpace = ctx.address_space
+
+    instruction_index: int = ii.instruction_index
+
+    if instruction_index in idx.string_reference_index_by_source_instruction_index:
+        # disassembler already identified string reference from instruction
+        return
+
+    reference_addresses: list[int] = []
+
+    if instruction_index in idx.data_reference_index_by_source_instruction_index:
+        for data_reference_index in idx.data_reference_index_by_source_instruction_index[instruction_index]:
+            data_reference: BinExport2.DataReference = be2.data_reference[data_reference_index]
+            data_reference_address: int = data_reference.address
+
+            if data_reference_address in idx.insn_address_by_index:
+                # appears to be code
+                continue
+
+            reference_addresses.append(data_reference_address)
+
+    for reference_address in reference_addresses:
+        try:
+            # if at end of segment then there might be an overrun here.
+            buf: bytes = address_space.read_memory(reference_address, 0x100)
+        except ReadMemoryError:
+            logger.debug("failed to read memory: 0x%x", reference_address)
+            continue
+
+        if capa.features.extractors.helpers.all_zeros(buf):
+            continue
+
+        is_string: bool = False
+
+        # note: we *always* break after the first iteration
+        for s in capa.features.extractors.strings.extract_ascii_strings(buf):
+            if s.offset != 0:
+                break
+
+            yield String(s.s), ih.address
+            is_string = True
+            break
+
+        # note: we *always* break after the first iteration
+        for s in capa.features.extractors.strings.extract_unicode_strings(buf):
+            if s.offset != 0:
+                break
+
+            yield String(s.s), ih.address
+            is_string = True
+            break
+
+        if not is_string:
+            yield Bytes(buf), ih.address
+
+
+def extract_insn_string_features(
+    fh: FunctionHandle, _bbh: BBHandle, ih: InsnHandle
+) -> Iterator[tuple[Feature, Address]]:
+    fhi: FunctionContext = fh.inner
+    ii: InstructionContext = ih.inner
+
+    be2: BinExport2 = fhi.ctx.be2
+    idx: BinExport2Index = fhi.ctx.idx
+
+    instruction_index: int = ii.instruction_index
+
+    if instruction_index in idx.string_reference_index_by_source_instruction_index:
+        for string_reference_index in idx.string_reference_index_by_source_instruction_index[instruction_index]:
+            string_reference: BinExport2.Reference = be2.string_reference[string_reference_index]
+            string_index: int = string_reference.string_table_index
+            string: str = be2.string_table[string_index]
+            yield String(string), ih.address
+
+
+def extract_insn_offset_features(
+    fh: FunctionHandle, bbh: BBHandle, ih: InsnHandle
+) -> Iterator[tuple[Feature, Address]]:
+    fhi: FunctionContext = fh.inner
+
+    if fhi.arch & HAS_ARCH_INTEL:
+        yield from capa.features.extractors.binexport2.arch.intel.insn.extract_insn_offset_features(fh, bbh, ih)
+    elif fhi.arch & HAS_ARCH_ARM:
+        yield from capa.features.extractors.binexport2.arch.arm.insn.extract_insn_offset_features(fh, bbh, ih)
+
+
+def extract_insn_nzxor_characteristic_features(
+    fh: FunctionHandle, bbh: BBHandle, ih: InsnHandle
+) -> Iterator[tuple[Feature, Address]]:
+    fhi: FunctionContext = fh.inner
+
+    if fhi.arch & HAS_ARCH_INTEL:
+        yield from capa.features.extractors.binexport2.arch.intel.insn.extract_insn_nzxor_characteristic_features(
+            fh, bbh, ih
+        )
+    elif fhi.arch & HAS_ARCH_ARM:
+        yield from capa.features.extractors.binexport2.arch.arm.insn.extract_insn_nzxor_characteristic_features(
+            fh, bbh, ih
+        )
+
+
+def extract_insn_mnemonic_features(
+    fh: FunctionHandle, bbh: BBHandle, ih: InsnHandle
+) -> Iterator[tuple[Feature, Address]]:
+    fhi: FunctionContext = fh.inner
+    ii: InstructionContext = ih.inner
+
+    be2: BinExport2 = fhi.ctx.be2
+
+    instruction: BinExport2.Instruction = be2.instruction[ii.instruction_index]
+    mnemonic: BinExport2.Mnemonic = be2.mnemonic[instruction.mnemonic_index]
+    mnemonic_name: str = mnemonic.name.lower()
+    yield Mnemonic(mnemonic_name), ih.address
+
+
+def extract_function_calls_from(fh: FunctionHandle, bbh: BBHandle, ih: InsnHandle) -> Iterator[tuple[Feature, Address]]:
+    """extract functions calls from features
+
+    most relevant at the function scope;
+    however, its most efficient to extract at the instruction scope.
+    """
+    fhi: FunctionContext = fh.inner
+    ii: InstructionContext = ih.inner
+
+    be2: BinExport2 = fhi.ctx.be2
+
+    instruction: BinExport2.Instruction = be2.instruction[ii.instruction_index]
+    for call_target_address in instruction.call_target:
+        addr: AbsoluteVirtualAddress = AbsoluteVirtualAddress(call_target_address)
+        yield Characteristic("calls from"), addr
+
+        if fh.address == addr:
+            yield Characteristic("recursive call"), addr
+
+
+def extract_function_indirect_call_characteristic_features(
+    fh: FunctionHandle, bbh: BBHandle, ih: InsnHandle
+) -> Iterator[tuple[Feature, Address]]:
+    fhi: FunctionContext = fh.inner
+
+    if fhi.arch & HAS_ARCH_INTEL:
+        yield from capa.features.extractors.binexport2.arch.intel.insn.extract_function_indirect_call_characteristic_features(
+            fh, bbh, ih
+        )
+    elif fhi.arch & HAS_ARCH_ARM:
+        yield from capa.features.extractors.binexport2.arch.arm.insn.extract_function_indirect_call_characteristic_features(
+            fh, bbh, ih
+        )
+
+
+def extract_features(f: FunctionHandle, bbh: BBHandle, insn: InsnHandle) -> Iterator[tuple[Feature, Address]]:
+    """extract instruction features"""
+    for inst_handler in INSTRUCTION_HANDLERS:
+        for feature, ea in inst_handler(f, bbh, insn):
+            yield feature, ea
+
+
+INSTRUCTION_HANDLERS = (
+    extract_insn_api_features,
+    extract_insn_number_features,
+    extract_insn_bytes_features,
+    extract_insn_string_features,
+    extract_insn_offset_features,
+    extract_insn_nzxor_characteristic_features,
+    extract_insn_mnemonic_features,
+    extract_function_calls_from,
+    extract_function_indirect_call_characteristic_features,
+)

capa/features/extractors/binja/basicblock.py

@@ -1,119 +1,36 @@
-# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
+# Copyright 2023 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# You may obtain a copy of the License at
-# Unless required by applicable law or agreed to in writing, software distributed under the License
+#
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#     http://www.apache.org/licenses/LICENSE-2.0
-# See the License for the specific language governing permissions and limitations under the License.
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
 
-import string
+from typing import Iterator
-from typing import Tuple, Iterator
 
-from binaryninja import Function
 from binaryninja import BasicBlock as BinjaBasicBlock
-from binaryninja import (
-    BinaryView,
-    SymbolType,
-    RegisterValueType,
-    VariableSourceType,
-    MediumLevelILOperation,
-    MediumLevelILBasicBlock,
-    MediumLevelILInstruction,
-)
 
 from capa.features.common import Feature, Characteristic
 from capa.features.address import Address
 from capa.features.basicblock import BasicBlock
-from capa.features.extractors.helpers import MIN_STACKSTRING_LEN
 from capa.features.extractors.base_extractor import BBHandle, FunctionHandle
 
 
-def get_printable_len_ascii(s: bytes) -> int:
+def extract_bb_tight_loop(fh: FunctionHandle, bbh: BBHandle) -> Iterator[tuple[Feature, Address]]:
-    """Return string length if all operand bytes are ascii or utf16-le printable"""
-    count = 0
-    for c in s:
-        if c == 0:
-            return count
-        if c < 127 and chr(c) in string.printable:
-            count += 1
-    return count
-
-
-def get_printable_len_wide(s: bytes) -> int:
-    """Return string length if all operand bytes are ascii or utf16-le printable"""
-    if all(c == 0x00 for c in s[1::2]):
-        return get_printable_len_ascii(s[::2])
-    return 0
-
-
-def get_stack_string_len(f: Function, il: MediumLevelILInstruction) -> int:
-    bv: BinaryView = f.view
-
-    if il.operation != MediumLevelILOperation.MLIL_CALL:
-        return 0
-
-    target = il.dest
-    if target.operation not in [MediumLevelILOperation.MLIL_CONST, MediumLevelILOperation.MLIL_CONST_PTR]:
-        return 0
-
-    addr = target.value.value
-    sym = bv.get_symbol_at(addr)
-    if not sym or sym.type not in [SymbolType.LibraryFunctionSymbol, SymbolType.SymbolicFunctionSymbol]:
-        return 0
-
-    if sym.name not in ["__builtin_strncpy", "__builtin_strcpy", "__builtin_wcscpy"]:
-        return 0
-
-    if len(il.params) < 2:
-        return 0
-
-    dest = il.params[0]
-    if dest.operation in [MediumLevelILOperation.MLIL_ADDRESS_OF, MediumLevelILOperation.MLIL_VAR]:
-        var = dest.src
-    else:
-        return 0
-
-    if var.source_type != VariableSourceType.StackVariableSourceType:
-        return 0
-
-    src = il.params[1]
-    if src.value.type != RegisterValueType.ConstantDataAggregateValue:
-        return 0
-
-    s = f.get_constant_data(RegisterValueType.ConstantDataAggregateValue, src.value.value)
-    return max(get_printable_len_ascii(bytes(s)), get_printable_len_wide(bytes(s)))
-
-
-def bb_contains_stackstring(f: Function, bb: MediumLevelILBasicBlock) -> bool:
-    """check basic block for stackstring indicators
-
-    true if basic block contains enough moves of constant bytes to the stack
-    """
-    count = 0
-    for il in bb:
-        count += get_stack_string_len(f, il)
-        if count > MIN_STACKSTRING_LEN:
-            return True
-
-    return False
-
-
-def extract_bb_stackstring(fh: FunctionHandle, bbh: BBHandle) -> Iterator[Tuple[Feature, Address]]:
-    """extract stackstring indicators from basic block"""
-    bb: Tuple[BinjaBasicBlock, MediumLevelILBasicBlock] = bbh.inner
-    if bb[1] is not None and bb_contains_stackstring(fh.inner, bb[1]):
-        yield Characteristic("stack string"), bbh.address
-
-
-def extract_bb_tight_loop(fh: FunctionHandle, bbh: BBHandle) -> Iterator[Tuple[Feature, Address]]:
     """extract tight loop indicators from a basic block"""
-    bb: Tuple[BinjaBasicBlock, MediumLevelILBasicBlock] = bbh.inner
+    bb: BinjaBasicBlock = bbh.inner
-    for edge in bb[0].outgoing_edges:
+    for edge in bb.outgoing_edges:
-        if edge.target.start == bb[0].start:
+        if edge.target.start == bb.start:
             yield Characteristic("tight loop"), bbh.address
 
 
-def extract_features(fh: FunctionHandle, bbh: BBHandle) -> Iterator[Tuple[Feature, Address]]:
+def extract_features(fh: FunctionHandle, bbh: BBHandle) -> Iterator[tuple[Feature, Address]]:
     """extract basic block features"""
     for bb_handler in BASIC_BLOCK_HANDLERS:
         for feature, addr in bb_handler(fh, bbh):
@@ -121,7 +38,4 @@ def extract_features(fh: FunctionHandle, bbh: BBHandle) -> Iterator[Tuple[Featur
     yield BasicBlock(), bbh.address
 
 
-BASIC_BLOCK_HANDLERS = (
+BASIC_BLOCK_HANDLERS = (extract_bb_tight_loop,)
-    extract_bb_tight_loop,
-    extract_bb_stackstring,
-)

capa/features/extractors/binja/extractor.py

@@ -1,11 +1,18 @@
-# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
+# Copyright 2023 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# You may obtain a copy of the License at
-# Unless required by applicable law or agreed to in writing, software distributed under the License
+#
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#     http://www.apache.org/licenses/LICENSE-2.0
-# See the License for the specific language governing permissions and limitations under the License.
+#
-from typing import List, Tuple, Iterator
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from typing import Iterator
 
 import binaryninja as binja
 
@@ -28,9 +35,9 @@ from capa.features.extractors.base_extractor import (
 
 class BinjaFeatureExtractor(StaticFeatureExtractor):
     def __init__(self, bv: binja.BinaryView):
-        super().__init__(hashes=SampleHashes.from_bytes(bv.file.raw.read(0, len(bv.file.raw))))
+        super().__init__(hashes=SampleHashes.from_bytes(bv.file.raw.read(0, bv.file.raw.length)))
         self.bv = bv
-        self.global_features: List[Tuple[Feature, Address]] = []
+        self.global_features: list[tuple[Feature, Address]] = []
         self.global_features.extend(capa.features.extractors.binja.file.extract_file_format(self.bv))
         self.global_features.extend(capa.features.extractors.binja.global_.extract_os(self.bv))
         self.global_features.extend(capa.features.extractors.binja.global_.extract_arch(self.bv))
@@ -48,31 +55,24 @@ class BinjaFeatureExtractor(StaticFeatureExtractor):
         for f in self.bv.functions:
             yield FunctionHandle(address=AbsoluteVirtualAddress(f.start), inner=f)
 
-    def extract_function_features(self, fh: FunctionHandle) -> Iterator[Tuple[Feature, Address]]:
+    def extract_function_features(self, fh: FunctionHandle) -> Iterator[tuple[Feature, Address]]:
         yield from capa.features.extractors.binja.function.extract_features(fh)
 
     def get_basic_blocks(self, fh: FunctionHandle) -> Iterator[BBHandle]:
         f: binja.Function = fh.inner
-        # Set up a MLIL basic block dict look up to associate the disassembly basic block with its MLIL basic block
-        mlil_lookup = {}
-        for mlil_bb in f.mlil.basic_blocks:
-            mlil_lookup[mlil_bb.source_block.start] = mlil_bb
-
         for bb in f.basic_blocks:
-            mlil_bb = mlil_lookup.get(bb.start)
+            yield BBHandle(address=AbsoluteVirtualAddress(bb.start), inner=bb)
 
-            yield BBHandle(address=AbsoluteVirtualAddress(bb.start), inner=(bb, mlil_bb))
+    def extract_basic_block_features(self, fh: FunctionHandle, bbh: BBHandle) -> Iterator[tuple[Feature, Address]]:
-
-    def extract_basic_block_features(self, fh: FunctionHandle, bbh: BBHandle) -> Iterator[Tuple[Feature, Address]]:
         yield from capa.features.extractors.binja.basicblock.extract_features(fh, bbh)
 
     def get_instructions(self, fh: FunctionHandle, bbh: BBHandle) -> Iterator[InsnHandle]:
         import capa.features.extractors.binja.helpers as binja_helpers
 
-        bb: Tuple[binja.BasicBlock, binja.MediumLevelILBasicBlock] = bbh.inner
+        bb: binja.BasicBlock = bbh.inner
-        addr = bb[0].start
+        addr = bb.start
 
-        for text, length in bb[0]:
+        for text, length in bb:
             insn = binja_helpers.DisassemblyInstruction(addr, length, text)
             yield InsnHandle(address=AbsoluteVirtualAddress(addr), inner=insn)
             addr += length

capa/features/extractors/binja/file.py

@@ -1,13 +1,18 @@
-# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
+# Copyright 2023 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# You may obtain a copy of the License at
-# Unless required by applicable law or agreed to in writing, software distributed under the License
+#
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#     http://www.apache.org/licenses/LICENSE-2.0
-# See the License for the specific language governing permissions and limitations under the License.
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
 
-import struct
+from typing import Iterator
-from typing import Tuple, Iterator
 
 from binaryninja import Segment, BinaryView, SymbolType, SymbolBinding
 
@@ -15,64 +20,42 @@ import capa.features.extractors.common
 import capa.features.extractors.helpers
 import capa.features.extractors.strings
 from capa.features.file import Export, Import, Section, FunctionName
-from capa.features.common import FORMAT_PE, FORMAT_ELF, Format, String, Feature, Characteristic
+from capa.features.common import (
+    FORMAT_PE,
+    FORMAT_ELF,
+    FORMAT_SC32,
+    FORMAT_SC64,
+    FORMAT_BINJA_DB,
+    Format,
+    String,
+    Feature,
+    Characteristic,
+)
 from capa.features.address import NO_ADDRESS, Address, FileOffsetAddress, AbsoluteVirtualAddress
 from capa.features.extractors.binja.helpers import read_c_string, unmangle_c_name
 
 
-def check_segment_for_pe(bv: BinaryView, seg: Segment) -> Iterator[Tuple[int, int]]:
+def check_segment_for_pe(bv: BinaryView, seg: Segment) -> Iterator[tuple[Feature, Address]]:
-    """check segment for embedded PE
+    """check segment for embedded PE"""
-
+    start = 0
-    adapted for binja from:
+    if bv.view_type == "PE" and seg.start == bv.start:
-    https://github.com/vivisect/vivisect/blob/7be4037b1cecc4551b397f840405a1fc606f9b53/PE/carve.py#L19
+        # If this is the first segment of the binary, skip the first bytes.
-    """
+        # Otherwise, there will always be a matched PE at the start of the binaryview.
-    mz_xor = [
-        (
-            capa.features.extractors.helpers.xor_static(b"MZ", i),
-            capa.features.extractors.helpers.xor_static(b"PE", i),
-            i,
-        )
-        for i in range(256)
-    ]
-
-    todo = []
-    # If this is the first segment of the binary, skip the first bytes. Otherwise, there will always be a matched
-    # PE at the start of the binaryview.
-    start = seg.start
-    if bv.view_type == "PE" and start == bv.start:
         start += 1
 
-    for mzx, pex, i in mz_xor:
+    buf = bv.read(seg.start, seg.length)
-        for off, _ in bv.find_all_data(start, seg.end, mzx):
-            todo.append((off, mzx, pex, i))
 
-    while len(todo):
+    for offset, _ in capa.features.extractors.helpers.carve_pe(buf, start):
-        off, mzx, pex, i = todo.pop()
+        yield Characteristic("embedded pe"), FileOffsetAddress(seg.start + offset)
-
-        # The MZ header has one field we will check e_lfanew is at 0x3c
-        e_lfanew = off + 0x3C
-
-        if seg.end < (e_lfanew + 4):
-            continue
-
-        newoff = struct.unpack("<I", capa.features.extractors.helpers.xor_static(bv.read(e_lfanew, 4), i))[0]
-
-        peoff = off + newoff
-        if seg.end < (peoff + 2):
-            continue
-
-        if bv.read(peoff, 2) == pex:
-            yield off, i
 
 
-def extract_file_embedded_pe(bv: BinaryView) -> Iterator[Tuple[Feature, Address]]:
+def extract_file_embedded_pe(bv: BinaryView) -> Iterator[tuple[Feature, Address]]:
     """extract embedded PE features"""
     for seg in bv.segments:
-        for ea, _ in check_segment_for_pe(bv, seg):
+        yield from check_segment_for_pe(bv, seg)
-            yield Characteristic("embedded pe"), FileOffsetAddress(ea)
 
 
-def extract_file_export_names(bv: BinaryView) -> Iterator[Tuple[Feature, Address]]:
+def extract_file_export_names(bv: BinaryView) -> Iterator[tuple[Feature, Address]]:
     """extract function exports"""
     for sym in bv.get_symbols_of_type(SymbolType.FunctionSymbol) + bv.get_symbols_of_type(SymbolType.DataSymbol):
         if sym.binding in [SymbolBinding.GlobalBinding, SymbolBinding.WeakBinding]:
@@ -106,7 +89,7 @@ def extract_file_export_names(bv: BinaryView) -> Iterator[Tuple[Feature, Address
         yield Characteristic("forwarded export"), AbsoluteVirtualAddress(sym.address)
 
 
-def extract_file_import_names(bv: BinaryView) -> Iterator[Tuple[Feature, Address]]:
+def extract_file_import_names(bv: BinaryView) -> Iterator[tuple[Feature, Address]]:
     """extract function imports
 
     1. imports by ordinal:
@@ -130,19 +113,19 @@ def extract_file_import_names(bv: BinaryView) -> Iterator[Tuple[Feature, Address
                 yield Import(name), addr
 
 
-def extract_file_section_names(bv: BinaryView) -> Iterator[Tuple[Feature, Address]]:
+def extract_file_section_names(bv: BinaryView) -> Iterator[tuple[Feature, Address]]:
     """extract section names"""
     for name, section in bv.sections.items():
         yield Section(name), AbsoluteVirtualAddress(section.start)
 
 
-def extract_file_strings(bv: BinaryView) -> Iterator[Tuple[Feature, Address]]:
+def extract_file_strings(bv: BinaryView) -> Iterator[tuple[Feature, Address]]:
     """extract ASCII and UTF-16 LE strings"""
     for s in bv.strings:
         yield String(s.value), FileOffsetAddress(s.start)
 
 
-def extract_file_function_names(bv: BinaryView) -> Iterator[Tuple[Feature, Address]]:
+def extract_file_function_names(bv: BinaryView) -> Iterator[tuple[Feature, Address]]:
     """
     extract the names of statically-linked library functions.
     """
@@ -161,12 +144,22 @@ def extract_file_function_names(bv: BinaryView) -> Iterator[Tuple[Feature, Addre
                 yield FunctionName(name[1:]), sym.address
 
 
-def extract_file_format(bv: BinaryView) -> Iterator[Tuple[Feature, Address]]:
+def extract_file_format(bv: BinaryView) -> Iterator[tuple[Feature, Address]]:
+    if bv.file.database is not None:
+        yield Format(FORMAT_BINJA_DB), NO_ADDRESS
+
     view_type = bv.view_type
     if view_type in ["PE", "COFF"]:
         yield Format(FORMAT_PE), NO_ADDRESS
     elif view_type == "ELF":
         yield Format(FORMAT_ELF), NO_ADDRESS
+    elif view_type == "Mapped":
+        if bv.arch.name == "x86":
+            yield Format(FORMAT_SC32), NO_ADDRESS
+        elif bv.arch.name == "x86_64":
+            yield Format(FORMAT_SC64), NO_ADDRESS
+        else:
+            raise NotImplementedError(f"unexpected raw file with arch: {bv.arch}")
     elif view_type == "Raw":
         # no file type to return when processing a binary file, but we want to continue processing
         return
@@ -174,7 +167,7 @@ def extract_file_format(bv: BinaryView) -> Iterator[Tuple[Feature, Address]]:
         raise NotImplementedError(f"unexpected file format: {view_type}")
 
 
-def extract_features(bv: BinaryView) -> Iterator[Tuple[Feature, Address]]:
+def extract_features(bv: BinaryView) -> Iterator[tuple[Feature, Address]]:
     """extract file features"""
     for file_handler in FILE_HANDLERS:
         for feature, addr in file_handler(bv):

capa/features/extractors/binja/find_binja_api.py

@@ -1,20 +1,35 @@
-# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
+# Copyright 2023 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# You may obtain a copy of the License at
-# Unless required by applicable law or agreed to in writing, software distributed under the License
+#
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#     http://www.apache.org/licenses/LICENSE-2.0
-# See the License for the specific language governing permissions and limitations under the License.
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import os
+import sys
+import logging
 import subprocess
+import importlib.util
+from typing import Optional
 from pathlib import Path
 
+logger = logging.getLogger(__name__)
+
+
 # When the script gets executed as a standalone executable (via PyInstaller), `import binaryninja` does not work because
 # we have excluded the binaryninja module in `pyinstaller.spec`. The trick here is to call the system Python and try
 # to find out the path of the binaryninja module that has been installed.
 # Note, including the binaryninja module in the `pyinstaller.spec` would not work, since the binaryninja module tries to
 # find the binaryninja core e.g., `libbinaryninjacore.dylib`, using a relative path. And this does not work when the
 # binaryninja module is extracted by the PyInstaller.
-code = r"""
+CODE = r"""
 from pathlib import Path
 from importlib import util
 spec = util.find_spec('binaryninja')
@@ -26,10 +41,146 @@ if spec is not None:
 """
 
 
-def find_binja_path() -> Path:
+def find_binaryninja_path_via_subprocess() -> Optional[Path]:
-    raw_output = subprocess.check_output(["python", "-c", code]).decode("ascii").strip()
+    raw_output = subprocess.check_output(["python", "-c", CODE]).decode("ascii").strip()
-    return Path(bytes.fromhex(raw_output).decode("utf8"))
+    output = bytes.fromhex(raw_output).decode("utf8")
+    if not output.strip():
+        return None
+    return Path(output)
+
+
+def get_desktop_entry(name: str) -> Optional[Path]:
+    """
+    Find the path for the given XDG Desktop Entry name.
+
+    Like:
+
+        >> get_desktop_entry("com.vector35.binaryninja.desktop")
+        Path("~/.local/share/applications/com.vector35.binaryninja.desktop")
+    """
+    assert sys.platform in ("linux", "linux2")
+    assert name.endswith(".desktop")
+
+    data_dirs = os.environ.get("XDG_DATA_DIRS", "/usr/share") + f":{Path.home()}/.local/share"
+    for data_dir in data_dirs.split(":"):
+        applications = Path(data_dir) / "applications"
+        for application in applications.glob("*.desktop"):
+            if application.name == name:
+                return application
+
+    return None
+
+
+def get_binaryninja_path(desktop_entry: Path) -> Optional[Path]:
+    # from: Exec=/home/wballenthin/software/binaryninja/binaryninja %u
+    # to:        /home/wballenthin/software/binaryninja/
+    for line in desktop_entry.read_text(encoding="utf-8").splitlines():
+        if not line.startswith("Exec="):
+            continue
+
+        if not line.endswith("binaryninja %u"):
+            continue
+
+        binaryninja_path = Path(line[len("Exec=") : -len("binaryninja %u")])
+        if not binaryninja_path.exists():
+            return None
+
+        return binaryninja_path
+
+    return None
+
+
+def validate_binaryninja_path(binaryninja_path: Path) -> bool:
+    if not binaryninja_path:
+        return False
+
+    module_path = binaryninja_path / "python"
+    if not module_path.is_dir():
+        return False
+
+    if not (module_path / "binaryninja" / "__init__.py").is_file():
+        return False
+
+    return True
+
+
+def find_binaryninja() -> Optional[Path]:
+    binaryninja_path = find_binaryninja_path_via_subprocess()
+    if not binaryninja_path or not validate_binaryninja_path(binaryninja_path):
+        if sys.platform == "linux" or sys.platform == "linux2":
+            # ok
+            logger.debug("detected OS: linux")
+        elif sys.platform == "darwin":
+            logger.warning("unsupported platform to find Binary Ninja: %s", sys.platform)
+            return None
+        elif sys.platform == "win32":
+            logger.warning("unsupported platform to find Binary Ninja: %s", sys.platform)
+            return None
+        else:
+            logger.warning("unsupported platform to find Binary Ninja: %s", sys.platform)
+            return None
+
+        desktop_entry = get_desktop_entry("com.vector35.binaryninja.desktop")
+        if not desktop_entry:
+            logger.debug("failed to find Binary Ninja application")
+            return None
+        logger.debug("found Binary Ninja application: %s", desktop_entry)
+
+        binaryninja_path = get_binaryninja_path(desktop_entry)
+        if not binaryninja_path:
+            logger.debug("failed to determine Binary Ninja installation path")
+            return None
+
+        if not validate_binaryninja_path(binaryninja_path):
+            logger.debug("failed to validate Binary Ninja installation")
+            return None
+
+    logger.debug("found Binary Ninja installation: %s", binaryninja_path)
+
+    return binaryninja_path / "python"
+
+
+def is_binaryninja_installed() -> bool:
+    """Is the binaryninja module ready to import?"""
+    try:
+        return importlib.util.find_spec("binaryninja") is not None
+    except ModuleNotFoundError:
+        return False
+
+
+def has_binaryninja() -> bool:
+    if is_binaryninja_installed():
+        logger.debug("found installed Binary Ninja API")
+        return True
+
+    logger.debug("Binary Ninja API not installed, searching...")
+
+    binaryninja_path = find_binaryninja()
+    if not binaryninja_path:
+        logger.debug("failed to find Binary Ninja installation")
+
+    logger.debug("found Binary Ninja API: %s", binaryninja_path)
+    return binaryninja_path is not None
+
+
+def load_binaryninja() -> bool:
+    try:
+        import binaryninja
+
+        return True
+    except ImportError:
+        binaryninja_path = find_binaryninja()
+        if not binaryninja_path:
+            return False
+
+        sys.path.append(binaryninja_path.absolute().as_posix())
+        try:
+            import binaryninja  # noqa: F401 unused import
+
+            return True
+        except ImportError:
+            return False
 
 
 if __name__ == "__main__":
-    print(find_binja_path())
+    print(find_binaryninja_path_via_subprocess())

capa/features/extractors/binja/function.py

@@ -1,18 +1,39 @@
-# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
+# Copyright 2023 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# You may obtain a copy of the License at
-# Unless required by applicable law or agreed to in writing, software distributed under the License
+#
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#     http://www.apache.org/licenses/LICENSE-2.0
-# See the License for the specific language governing permissions and limitations under the License.
+#
-from typing import Tuple, Iterator
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
 
-from binaryninja import Function, BinaryView, SymbolType, RegisterValueType, LowLevelILOperation
+import string
+from typing import Iterator
+
+from binaryninja import (
+    Function,
+    BinaryView,
+    SymbolType,
+    ILException,
+    RegisterValueType,
+    VariableSourceType,
+    LowLevelILOperation,
+    MediumLevelILOperation,
+    MediumLevelILBasicBlock,
+    MediumLevelILInstruction,
+)
 
 from capa.features.file import FunctionName
 from capa.features.common import Feature, Characteristic
 from capa.features.address import Address, AbsoluteVirtualAddress
 from capa.features.extractors import loops
+from capa.features.extractors.helpers import MIN_STACKSTRING_LEN
+from capa.features.extractors.binja.helpers import get_llil_instr_at_addr
 from capa.features.extractors.base_extractor import FunctionHandle
 
 
@@ -24,7 +45,7 @@ def extract_function_calls_to(fh: FunctionHandle):
         # Everything that is a code reference to the current function is considered a caller, which actually includes
         # many other references that are NOT a caller. For example, an instruction `push function_start` will also be
         # considered a caller to the function
-        llil = caller.llil
+        llil = get_llil_instr_at_addr(func.view, caller.address)
         if (llil is None) or llil.operation not in [
             LowLevelILOperation.LLIL_CALL,
             LowLevelILOperation.LLIL_CALL_STACK_ADJUST,
@@ -33,14 +54,13 @@ def extract_function_calls_to(fh: FunctionHandle):
         ]:
             continue
 
-        if llil.dest.value.type not in [
+        if llil.dest.operation not in [
-            RegisterValueType.ImportedAddressValue,
+            LowLevelILOperation.LLIL_CONST,
-            RegisterValueType.ConstantValue,
+            LowLevelILOperation.LLIL_CONST_PTR,
-            RegisterValueType.ConstantPointerValue,
         ]:
             continue
 
-        address = llil.dest.value.value
+        address = llil.dest.constant
         if address != func.start:
             continue
 
@@ -95,10 +115,103 @@ def extract_function_name(fh: FunctionHandle):
             yield FunctionName(name[1:]), sym.address
 
 
-def extract_features(fh: FunctionHandle) -> Iterator[Tuple[Feature, Address]]:
+def get_printable_len_ascii(s: bytes) -> int:
+    """Return string length if all operand bytes are ascii or utf16-le printable"""
+    count = 0
+    for c in s:
+        if c == 0:
+            return count
+        if c < 127 and chr(c) in string.printable:
+            count += 1
+    return count
+
+
+def get_printable_len_wide(s: bytes) -> int:
+    """Return string length if all operand bytes are ascii or utf16-le printable"""
+    if all(c == 0x00 for c in s[1::2]):
+        return get_printable_len_ascii(s[::2])
+    return 0
+
+
+def get_stack_string_len(f: Function, il: MediumLevelILInstruction) -> int:
+    bv: BinaryView = f.view
+
+    if il.operation != MediumLevelILOperation.MLIL_CALL:
+        return 0
+
+    target = il.dest
+    if target.operation not in [MediumLevelILOperation.MLIL_CONST, MediumLevelILOperation.MLIL_CONST_PTR]:
+        return 0
+
+    addr = target.value.value
+    sym = bv.get_symbol_at(addr)
+    if not sym or sym.type not in [SymbolType.LibraryFunctionSymbol, SymbolType.SymbolicFunctionSymbol]:
+        return 0
+
+    if sym.name not in ["__builtin_strncpy", "__builtin_strcpy", "__builtin_wcscpy"]:
+        return 0
+
+    if len(il.params) < 2:
+        return 0
+
+    dest = il.params[0]
+    if dest.operation in [MediumLevelILOperation.MLIL_ADDRESS_OF, MediumLevelILOperation.MLIL_VAR]:
+        var = dest.src
+    else:
+        return 0
+
+    if var.source_type != VariableSourceType.StackVariableSourceType:
+        return 0
+
+    src = il.params[1]
+    if src.value.type != RegisterValueType.ConstantDataAggregateValue:
+        return 0
+
+    s = f.get_constant_data(RegisterValueType.ConstantDataAggregateValue, src.value.value)
+    return max(get_printable_len_ascii(bytes(s)), get_printable_len_wide(bytes(s)))
+
+
+def bb_contains_stackstring(f: Function, bb: MediumLevelILBasicBlock) -> bool:
+    """check basic block for stackstring indicators
+
+    true if basic block contains enough moves of constant bytes to the stack
+    """
+    count = 0
+    for il in bb:
+        count += get_stack_string_len(f, il)
+        if count > MIN_STACKSTRING_LEN:
+            return True
+
+    return False
+
+
+def extract_stackstring(fh: FunctionHandle):
+    """extract stackstring indicators"""
+    func: Function = fh.inner
+    bv: BinaryView = func.view
+    if bv is None:
+        return
+
+    try:
+        mlil = func.mlil
+    except ILException:
+        return
+
+    for block in mlil.basic_blocks:
+        if bb_contains_stackstring(func, block):
+            yield Characteristic("stack string"), block.source_block.start
+
+
+def extract_features(fh: FunctionHandle) -> Iterator[tuple[Feature, Address]]:
     for func_handler in FUNCTION_HANDLERS:
         for feature, addr in func_handler(fh):
             yield feature, addr
 
 
-FUNCTION_HANDLERS = (extract_function_calls_to, extract_function_loop, extract_recursive_call, extract_function_name)
+FUNCTION_HANDLERS = (
+    extract_function_calls_to,
+    extract_function_loop,
+    extract_recursive_call,
+    extract_function_name,
+    extract_stackstring,
+)

capa/features/extractors/binja/global_.py

@@ -1,12 +1,19 @@
-# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
+# Copyright 2023 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# You may obtain a copy of the License at
-# Unless required by applicable law or agreed to in writing, software distributed under the License
+#
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#     http://www.apache.org/licenses/LICENSE-2.0
-# See the License for the specific language governing permissions and limitations under the License.
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 import logging
-from typing import Tuple, Iterator
+from typing import Iterator
 
 from binaryninja import BinaryView
 
@@ -16,7 +23,7 @@ from capa.features.address import NO_ADDRESS, Address
 logger = logging.getLogger(__name__)
 
 
-def extract_os(bv: BinaryView) -> Iterator[Tuple[Feature, Address]]:
+def extract_os(bv: BinaryView) -> Iterator[tuple[Feature, Address]]:
     name = bv.platform.name
     if "-" in name:
         name = name.split("-")[0]
@@ -45,7 +52,7 @@ def extract_os(bv: BinaryView) -> Iterator[Tuple[Feature, Address]]:
         return
 
 
-def extract_arch(bv: BinaryView) -> Iterator[Tuple[Feature, Address]]:
+def extract_arch(bv: BinaryView) -> Iterator[tuple[Feature, Address]]:
     arch = bv.arch.name
     if arch == "x86_64":
         yield Arch(ARCH_AMD64), NO_ADDRESS

capa/features/extractors/binja/helpers.py

@@ -1,15 +1,22 @@
-# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
+# Copyright 2023 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# You may obtain a copy of the License at
-# Unless required by applicable law or agreed to in writing, software distributed under the License
+#
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#     http://www.apache.org/licenses/LICENSE-2.0
-# See the License for the specific language governing permissions and limitations under the License.
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 import re
-from typing import List, Callable
+from typing import Callable, Optional
 from dataclasses import dataclass
 
-from binaryninja import BinaryView, LowLevelILInstruction
+from binaryninja import BinaryView, LowLevelILFunction, LowLevelILInstruction
 from binaryninja.architecture import InstructionTextToken
 
 
@@ -17,7 +24,7 @@ from binaryninja.architecture import InstructionTextToken
 class DisassemblyInstruction:
     address: int
     length: int
-    text: List[InstructionTextToken]
+    text: list[InstructionTextToken]
 
 
 LLIL_VISITOR = Callable[[LowLevelILInstruction, LowLevelILInstruction, int], bool]
@@ -54,7 +61,7 @@ def unmangle_c_name(name: str) -> str:
 
 
 def read_c_string(bv: BinaryView, offset: int, max_len: int) -> str:
-    s: List[str] = []
+    s: list[str] = []
     while len(s) < max_len:
         try:
             c = bv.read(offset + len(s), 1)[0]
@@ -67,3 +74,13 @@ def read_c_string(bv: BinaryView, offset: int, max_len: int) -> str:
         s.append(chr(c))
 
     return "".join(s)
+
+
+def get_llil_instr_at_addr(bv: BinaryView, addr: int) -> Optional[LowLevelILInstruction]:
+    arch = bv.arch
+    buffer = bv.read(addr, arch.max_instr_length)
+    llil = LowLevelILFunction(arch=arch)
+    llil.current_address = addr
+    if arch.get_instruction_low_level_il(buffer, addr, llil) == 0:
+        return None
+    return llil[0]

capa/features/extractors/binja/insn.py

@@ -1,15 +1,22 @@
-# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
+# Copyright 2023 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# You may obtain a copy of the License at
-# Unless required by applicable law or agreed to in writing, software distributed under the License
+#
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#     http://www.apache.org/licenses/LICENSE-2.0
-# See the License for the specific language governing permissions and limitations under the License.
+#
-from typing import Any, List, Tuple, Iterator, Optional
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+from typing import Any, Optional
+from collections.abc import Iterator
 
-from binaryninja import Function
+import binaryninja as bn
-from binaryninja import BasicBlock as BinjaBasicBlock
 from binaryninja import (
+    Function,
     BinaryView,
     ILRegister,
     SymbolType,
@@ -23,7 +30,7 @@ import capa.features.extractors.helpers
 from capa.features.insn import API, MAX_STRUCTURE_SIZE, Number, Offset, Mnemonic, OperandNumber, OperandOffset
 from capa.features.common import MAX_BYTES_FEATURE_SIZE, Bytes, String, Feature, Characteristic
 from capa.features.address import Address, AbsoluteVirtualAddress
-from capa.features.extractors.binja.helpers import DisassemblyInstruction, visit_llil_exprs
+from capa.features.extractors.binja.helpers import DisassemblyInstruction, visit_llil_exprs, get_llil_instr_at_addr
 from capa.features.extractors.base_extractor import BBHandle, InsnHandle, FunctionHandle
 
 # security cookie checks may perform non-zeroing XORs, these are expected within a certain
@@ -36,35 +43,27 @@ SECURITY_COOKIE_BYTES_DELTA = 0x40
 # 2. The function must only make one call/jump to another address
 # If the function being checked is a stub function, returns the target address. Otherwise, return None.
 def is_stub_function(bv: BinaryView, addr: int) -> Optional[int]:
-    funcs = bv.get_functions_at(addr)
+    llil = get_llil_instr_at_addr(bv, addr)
-    for func in funcs:
+    if llil is None or llil.operation not in [
-        if len(func.basic_blocks) != 1:
-            continue
-
-        call_count = 0
-        call_target = None
-        for il in func.llil.instructions:
-            if il.operation in [
         LowLevelILOperation.LLIL_CALL,
         LowLevelILOperation.LLIL_CALL_STACK_ADJUST,
         LowLevelILOperation.LLIL_JUMP,
         LowLevelILOperation.LLIL_TAILCALL,
     ]:
-                call_count += 1
-                if il.dest.value.type in [
-                    RegisterValueType.ImportedAddressValue,
-                    RegisterValueType.ConstantValue,
-                    RegisterValueType.ConstantPointerValue,
-                ]:
-                    call_target = il.dest.value.value
-
-        if call_count == 1 and call_target is not None:
-            return call_target
-
         return None
 
+    # The LLIL instruction retrieved by `get_llil_instr_at_addr` did not go through a full analysis, so we cannot check
+    # `llil.dest.value.type` here
+    if llil.dest.operation not in [
+        LowLevelILOperation.LLIL_CONST,
+        LowLevelILOperation.LLIL_CONST_PTR,
+    ]:
+        return None
 
-def extract_insn_api_features(fh: FunctionHandle, bbh: BBHandle, ih: InsnHandle) -> Iterator[Tuple[Feature, Address]]:
+    return llil.dest.constant
+
+
+def extract_insn_api_features(fh: FunctionHandle, bbh: BBHandle, ih: InsnHandle) -> Iterator[tuple[Feature, Address]]:
     """
     parse instruction API features
 
@@ -123,7 +122,7 @@ def extract_insn_api_features(fh: FunctionHandle, bbh: BBHandle, ih: InsnHandle)
 
 def extract_insn_number_features(
     fh: FunctionHandle, bbh: BBHandle, ih: InsnHandle
-) -> Iterator[Tuple[Feature, Address]]:
+) -> Iterator[tuple[Feature, Address]]:
     """
     parse instruction number features
     example:
@@ -131,7 +130,7 @@ def extract_insn_number_features(
     """
     func: Function = fh.inner
 
-    results: List[Tuple[Any[Number, OperandNumber], Address]] = []
+    results: list[tuple[Any[Number, OperandNumber], Address]] = []
 
     def llil_checker(il: LowLevelILInstruction, parent: LowLevelILInstruction, index: int) -> bool:
         if il.operation == LowLevelILOperation.LLIL_LOAD:
@@ -162,7 +161,7 @@ def extract_insn_number_features(
     yield from results
 
 
-def extract_insn_bytes_features(fh: FunctionHandle, bbh: BBHandle, ih: InsnHandle) -> Iterator[Tuple[Feature, Address]]:
+def extract_insn_bytes_features(fh: FunctionHandle, bbh: BBHandle, ih: InsnHandle) -> Iterator[tuple[Feature, Address]]:
     """
     parse referenced byte sequences
     example:
@@ -209,7 +208,7 @@ def extract_insn_bytes_features(fh: FunctionHandle, bbh: BBHandle, ih: InsnHandl
 
 def extract_insn_string_features(
     fh: FunctionHandle, bbh: BBHandle, ih: InsnHandle
-) -> Iterator[Tuple[Feature, Address]]:
+) -> Iterator[tuple[Feature, Address]]:
     """
     parse instruction string features
 
@@ -266,7 +265,7 @@ def extract_insn_string_features(
 
 def extract_insn_offset_features(
     fh: FunctionHandle, bbh: BBHandle, ih: InsnHandle
-) -> Iterator[Tuple[Feature, Address]]:
+) -> Iterator[tuple[Feature, Address]]:
     """
     parse instruction structure offset features
 
@@ -275,7 +274,7 @@ def extract_insn_offset_features(
     """
     func: Function = fh.inner
 
-    results: List[Tuple[Any[Offset, OperandOffset], Address]] = []
+    results: list[tuple[Any[Offset, OperandOffset], Address]] = []
     address_size = func.view.arch.address_size * 8
 
     def llil_checker(il: LowLevelILInstruction, parent: LowLevelILInstruction, index: int) -> bool:
@@ -324,7 +323,7 @@ def extract_insn_offset_features(
     yield from results
 
 
-def is_nzxor_stack_cookie(f: Function, bb: BinjaBasicBlock, llil: LowLevelILInstruction) -> bool:
+def is_nzxor_stack_cookie(f: Function, bb: bn.BasicBlock, llil: LowLevelILInstruction) -> bool:
     """check if nzxor exists within stack cookie delta"""
     # TODO(xusheng): use LLIL SSA to do more accurate analysis
     # https://github.com/mandiant/capa/issues/1609
@@ -353,7 +352,7 @@ def is_nzxor_stack_cookie(f: Function, bb: BinjaBasicBlock, llil: LowLevelILInst
 
 def extract_insn_nzxor_characteristic_features(
     fh: FunctionHandle, bbh: BBHandle, ih: InsnHandle
-) -> Iterator[Tuple[Feature, Address]]:
+) -> Iterator[tuple[Feature, Address]]:
     """
     parse instruction non-zeroing XOR instruction
     ignore expected non-zeroing XORs, e.g. security cookies
@@ -367,7 +366,7 @@ def extract_insn_nzxor_characteristic_features(
         # e.g., <llil: eax = 0>, (LLIL_SET_REG). So we do not need to check whether the two operands are the same.
         if il.operation == LowLevelILOperation.LLIL_XOR:
             # Exclude cases related to the stack cookie
-            if is_nzxor_stack_cookie(fh.inner, bbh.inner[0], il):
+            if is_nzxor_stack_cookie(fh.inner, bbh.inner, il):
                 return False
             results.append((Characteristic("nzxor"), ih.address))
             return False
@@ -382,7 +381,7 @@ def extract_insn_nzxor_characteristic_features(
 
 def extract_insn_mnemonic_features(
     fh: FunctionHandle, bbh: BBHandle, ih: InsnHandle
-) -> Iterator[Tuple[Feature, Address]]:
+) -> Iterator[tuple[Feature, Address]]:
     """parse instruction mnemonic features"""
     insn: DisassemblyInstruction = ih.inner
     yield Mnemonic(insn.text[0].text), ih.address
@@ -390,7 +389,7 @@ def extract_insn_mnemonic_features(
 
 def extract_insn_obfs_call_plus_5_characteristic_features(
     fh: FunctionHandle, bbh: BBHandle, ih: InsnHandle
-) -> Iterator[Tuple[Feature, Address]]:
+) -> Iterator[tuple[Feature, Address]]:
     """
     parse call $+5 instruction from the given instruction.
     """
@@ -401,7 +400,7 @@ def extract_insn_obfs_call_plus_5_characteristic_features(
 
 def extract_insn_peb_access_characteristic_features(
     fh: FunctionHandle, bbh: BBHandle, ih: InsnHandle
-) -> Iterator[Tuple[Feature, Address]]:
+) -> Iterator[tuple[Feature, Address]]:
     """parse instruction peb access
 
     fs:[0x30] on x86, gs:[0x60] on x64
@@ -444,7 +443,7 @@ def extract_insn_peb_access_characteristic_features(
 
 def extract_insn_segment_access_features(
     fh: FunctionHandle, bbh: BBHandle, ih: InsnHandle
-) -> Iterator[Tuple[Feature, Address]]:
+) -> Iterator[tuple[Feature, Address]]:
     """parse instruction fs or gs access"""
     func: Function = fh.inner
 
@@ -471,7 +470,7 @@ def extract_insn_segment_access_features(
 
 def extract_insn_cross_section_cflow(
     fh: FunctionHandle, bbh: BBHandle, ih: InsnHandle
-) -> Iterator[Tuple[Feature, Address]]:
+) -> Iterator[tuple[Feature, Address]]:
     """inspect the instruction for a CALL or JMP that crosses section boundaries"""
     func: Function = fh.inner
     bv: BinaryView = func.view
@@ -491,7 +490,7 @@ def extract_insn_cross_section_cflow(
             yield Characteristic("cross section flow"), ih.address
 
 
-def extract_function_calls_from(fh: FunctionHandle, bbh: BBHandle, ih: InsnHandle) -> Iterator[Tuple[Feature, Address]]:
+def extract_function_calls_from(fh: FunctionHandle, bbh: BBHandle, ih: InsnHandle) -> Iterator[tuple[Feature, Address]]:
     """extract functions calls from features
 
     most relevant at the function scope, however, its most efficient to extract at the instruction scope
@@ -534,7 +533,7 @@ def extract_function_calls_from(fh: FunctionHandle, bbh: BBHandle, ih: InsnHandl
 
 def extract_function_indirect_call_characteristic_features(
     fh: FunctionHandle, bbh: BBHandle, ih: InsnHandle
-) -> Iterator[Tuple[Feature, Address]]:
+) -> Iterator[tuple[Feature, Address]]:
     """extract indirect function calls (e.g., call eax or call dword ptr [edx+4])
     does not include calls like => call ds:dword_ABD4974
 
@@ -562,7 +561,7 @@ def extract_function_indirect_call_characteristic_features(
     yield Characteristic("indirect call"), ih.address
 
 
-def extract_features(f: FunctionHandle, bbh: BBHandle, insn: InsnHandle) -> Iterator[Tuple[Feature, Address]]:
+def extract_features(f: FunctionHandle, bbh: BBHandle, insn: InsnHandle) -> Iterator[tuple[Feature, Address]]:
     """extract instruction features"""
     for inst_handler in INSTRUCTION_HANDLERS:
         for feature, ea in inst_handler(f, bbh, insn):

capa/features/extractors/cape/call.py

@@ -1,14 +1,22 @@
-# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
+# Copyright 2023 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# You may obtain a copy of the License at
-# Unless required by applicable law or agreed to in writing, software distributed under the License
+#
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#     http://www.apache.org/licenses/LICENSE-2.0
-# See the License for the specific language governing permissions and limitations under the License.
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 
 import logging
-from typing import Tuple, Iterator
+from typing import Iterator
 
+import capa.features.extractors.helpers
 from capa.helpers import assert_never
 from capa.features.insn import API, Number
 from capa.features.common import String, Feature
@@ -19,7 +27,7 @@ from capa.features.extractors.base_extractor import CallHandle, ThreadHandle, Pr
 logger = logging.getLogger(__name__)
 
 
-def extract_call_features(ph: ProcessHandle, th: ThreadHandle, ch: CallHandle) -> Iterator[Tuple[Feature, Address]]:
+def extract_call_features(ph: ProcessHandle, th: ThreadHandle, ch: CallHandle) -> Iterator[tuple[Feature, Address]]:
     """
     this method extracts the given call's features (such as API name and arguments),
     and returns them as API, Number, and String features.
@@ -50,10 +58,11 @@ def extract_call_features(ph: ProcessHandle, th: ThreadHandle, ch: CallHandle) -
         else:
             assert_never(value)
 
-    yield API(call.api), ch.address
+    for name in capa.features.extractors.helpers.generate_symbols("", call.api):
+        yield API(name), ch.address
 
 
-def extract_features(ph: ProcessHandle, th: ThreadHandle, ch: CallHandle) -> Iterator[Tuple[Feature, Address]]:
+def extract_features(ph: ProcessHandle, th: ThreadHandle, ch: CallHandle) -> Iterator[tuple[Feature, Address]]:
     for handler in CALL_HANDLERS:
         for feature, addr in handler(ph, th, ch):
             yield feature, addr

capa/features/extractors/cape/extractor.py

@@ -1,13 +1,20 @@
-# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
+# Copyright 2023 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# You may obtain a copy of the License at
-# Unless required by applicable law or agreed to in writing, software distributed under the License
+#
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#     http://www.apache.org/licenses/LICENSE-2.0
-# See the License for the specific language governing permissions and limitations under the License.
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 
 import logging
-from typing import Dict, Tuple, Union, Iterator
+from typing import Union, Iterator
 
 import capa.features.extractors.cape.call
 import capa.features.extractors.cape.file
@@ -15,8 +22,8 @@ import capa.features.extractors.cape.thread
 import capa.features.extractors.cape.global_
 import capa.features.extractors.cape.process
 from capa.exceptions import EmptyReportError, UnsupportedFormatError
-from capa.features.common import Feature, Characteristic
+from capa.features.common import Feature
-from capa.features.address import NO_ADDRESS, Address, AbsoluteVirtualAddress, _NoAddress
+from capa.features.address import Address, AbsoluteVirtualAddress, _NoAddress
 from capa.features.extractors.cape.models import Call, Static, Process, CapeReport
 from capa.features.extractors.base_extractor import (
     CallHandle,
@@ -47,19 +54,20 @@ class CapeExtractor(DynamicFeatureExtractor):
 
     def get_base_address(self) -> Union[AbsoluteVirtualAddress, _NoAddress, None]:
         # value according to the PE header, the actual trace may use a different imagebase
-        assert self.report.static is not None and self.report.static.pe is not None
+        assert self.report.static is not None
+        assert self.report.static.pe is not None
         return AbsoluteVirtualAddress(self.report.static.pe.imagebase)
 
-    def extract_global_features(self) -> Iterator[Tuple[Feature, Address]]:
+    def extract_global_features(self) -> Iterator[tuple[Feature, Address]]:
         yield from self.global_features
 
-    def extract_file_features(self) -> Iterator[Tuple[Feature, Address]]:
+    def extract_file_features(self) -> Iterator[tuple[Feature, Address]]:
         yield from capa.features.extractors.cape.file.extract_features(self.report)
 
     def get_processes(self) -> Iterator[ProcessHandle]:
         yield from capa.features.extractors.cape.file.get_processes(self.report)
 
-    def extract_process_features(self, ph: ProcessHandle) -> Iterator[Tuple[Feature, Address]]:
+    def extract_process_features(self, ph: ProcessHandle) -> Iterator[tuple[Feature, Address]]:
         yield from capa.features.extractors.cape.process.extract_features(ph)
 
     def get_process_name(self, ph) -> str:
@@ -69,19 +77,15 @@ class CapeExtractor(DynamicFeatureExtractor):
     def get_threads(self, ph: ProcessHandle) -> Iterator[ThreadHandle]:
         yield from capa.features.extractors.cape.process.get_threads(ph)
 
-    def extract_thread_features(self, ph: ProcessHandle, th: ThreadHandle) -> Iterator[Tuple[Feature, Address]]:
+    def extract_thread_features(self, ph: ProcessHandle, th: ThreadHandle) -> Iterator[tuple[Feature, Address]]:
-        if False:
+        yield from []
-            # force this routine to be a generator,
-            # but we don't actually have any elements to generate.
-            yield Characteristic("never"), NO_ADDRESS
-        return
 
     def get_calls(self, ph: ProcessHandle, th: ThreadHandle) -> Iterator[CallHandle]:
         yield from capa.features.extractors.cape.thread.get_calls(ph, th)
 
     def extract_call_features(
         self, ph: ProcessHandle, th: ThreadHandle, ch: CallHandle
-    ) -> Iterator[Tuple[Feature, Address]]:
+    ) -> Iterator[tuple[Feature, Address]]:
         yield from capa.features.extractors.cape.call.extract_features(ph, th, ch)
 
     def get_call_name(self, ph, th, ch) -> str:
@@ -122,7 +126,7 @@ class CapeExtractor(DynamicFeatureExtractor):
         return "".join(parts)
 
     @classmethod
-    def from_report(cls, report: Dict) -> "CapeExtractor":
+    def from_report(cls, report: dict) -> "CapeExtractor":
         cr = CapeReport.model_validate(report)
 
         if cr.info.version not in TESTED_VERSIONS:

capa/features/extractors/cape/file.py

@@ -1,13 +1,20 @@
-# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
+# Copyright 2023 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# You may obtain a copy of the License at
-# Unless required by applicable law or agreed to in writing, software distributed under the License
+#
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#     http://www.apache.org/licenses/LICENSE-2.0
-# See the License for the specific language governing permissions and limitations under the License.
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 
 import logging
-from typing import Tuple, Iterator
+from typing import Iterator
 
 from capa.features.file import Export, Import, Section
 from capa.features.common import String, Feature
@@ -41,7 +48,7 @@ def get_processes(report: CapeReport) -> Iterator[ProcessHandle]:
             seen_processes[addr].append(process)
 
 
-def extract_import_names(report: CapeReport) -> Iterator[Tuple[Feature, Address]]:
+def extract_import_names(report: CapeReport) -> Iterator[tuple[Feature, Address]]:
     """
     extract imported function names
     """
@@ -62,57 +69,75 @@ def extract_import_names(report: CapeReport) -> Iterator[Tuple[Feature, Address]
                 yield Import(name), AbsoluteVirtualAddress(function.address)
 
 
-def extract_export_names(report: CapeReport) -> Iterator[Tuple[Feature, Address]]:
+def extract_export_names(report: CapeReport) -> Iterator[tuple[Feature, Address]]:
     assert report.static is not None and report.static.pe is not None
     for function in report.static.pe.exports:
         yield Export(function.name), AbsoluteVirtualAddress(function.address)
 
 
-def extract_section_names(report: CapeReport) -> Iterator[Tuple[Feature, Address]]:
+def extract_section_names(report: CapeReport) -> Iterator[tuple[Feature, Address]]:
     assert report.static is not None and report.static.pe is not None
     for section in report.static.pe.sections:
         yield Section(section.name), AbsoluteVirtualAddress(section.virtual_address)
 
 
-def extract_file_strings(report: CapeReport) -> Iterator[Tuple[Feature, Address]]:
+def extract_file_strings(report: CapeReport) -> Iterator[tuple[Feature, Address]]:
     if report.strings is not None:
         for string in report.strings:
             yield String(string), NO_ADDRESS
 
 
-def extract_used_regkeys(report: CapeReport) -> Iterator[Tuple[Feature, Address]]:
+def extract_used_regkeys(report: CapeReport) -> Iterator[tuple[Feature, Address]]:
+    if not report.behavior.summary:
+        return
+
     for regkey in report.behavior.summary.keys:
         yield String(regkey), NO_ADDRESS
 
 
-def extract_used_files(report: CapeReport) -> Iterator[Tuple[Feature, Address]]:
+def extract_used_files(report: CapeReport) -> Iterator[tuple[Feature, Address]]:
+    if not report.behavior.summary:
+        return
+
     for file in report.behavior.summary.files:
         yield String(file), NO_ADDRESS
 
 
-def extract_used_mutexes(report: CapeReport) -> Iterator[Tuple[Feature, Address]]:
+def extract_used_mutexes(report: CapeReport) -> Iterator[tuple[Feature, Address]]:
+    if not report.behavior.summary:
+        return
+
     for mutex in report.behavior.summary.mutexes:
         yield String(mutex), NO_ADDRESS
 
 
-def extract_used_commands(report: CapeReport) -> Iterator[Tuple[Feature, Address]]:
+def extract_used_commands(report: CapeReport) -> Iterator[tuple[Feature, Address]]:
+    if not report.behavior.summary:
+        return
+
     for cmd in report.behavior.summary.executed_commands:
         yield String(cmd), NO_ADDRESS
 
 
-def extract_used_apis(report: CapeReport) -> Iterator[Tuple[Feature, Address]]:
+def extract_used_apis(report: CapeReport) -> Iterator[tuple[Feature, Address]]:
+    if not report.behavior.summary:
+        return
+
     for symbol in report.behavior.summary.resolved_apis:
         yield String(symbol), NO_ADDRESS
 
 
-def extract_used_services(report: CapeReport) -> Iterator[Tuple[Feature, Address]]:
+def extract_used_services(report: CapeReport) -> Iterator[tuple[Feature, Address]]:
+    if not report.behavior.summary:
+        return
+
     for svc in report.behavior.summary.created_services:
         yield String(svc), NO_ADDRESS
     for svc in report.behavior.summary.started_services:
         yield String(svc), NO_ADDRESS
 
 
-def extract_features(report: CapeReport) -> Iterator[Tuple[Feature, Address]]:
+def extract_features(report: CapeReport) -> Iterator[tuple[Feature, Address]]:
     for handler in FILE_HANDLERS:
         for feature, addr in handler(report):
             yield feature, addr

capa/features/extractors/cape/global_.py

@@ -1,13 +1,20 @@
-# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
+# Copyright 2023 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# You may obtain a copy of the License at
-# Unless required by applicable law or agreed to in writing, software distributed under the License
+#
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#     http://www.apache.org/licenses/LICENSE-2.0
-# See the License for the specific language governing permissions and limitations under the License.
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 
 import logging
-from typing import Tuple, Iterator
+from typing import Iterator
 
 from capa.features.common import (
     OS,
@@ -28,7 +35,7 @@ from capa.features.extractors.cape.models import CapeReport
 logger = logging.getLogger(__name__)
 
 
-def extract_arch(report: CapeReport) -> Iterator[Tuple[Feature, Address]]:
+def extract_arch(report: CapeReport) -> Iterator[tuple[Feature, Address]]:
     if "Intel 80386" in report.target.file.type:
         yield Arch(ARCH_I386), NO_ADDRESS
     elif "x86-64" in report.target.file.type:
@@ -40,7 +47,7 @@ def extract_arch(report: CapeReport) -> Iterator[Tuple[Feature, Address]]:
         )
 
 
-def extract_format(report: CapeReport) -> Iterator[Tuple[Feature, Address]]:
+def extract_format(report: CapeReport) -> Iterator[tuple[Feature, Address]]:
     if "PE" in report.target.file.type:
         yield Format(FORMAT_PE), NO_ADDRESS
     elif "ELF" in report.target.file.type:
@@ -48,11 +55,11 @@ def extract_format(report: CapeReport) -> Iterator[Tuple[Feature, Address]]:
     else:
         logger.warning("unknown file format, file command output: %s", report.target.file.type)
         raise ValueError(
-            "unrecognized file format from the CAPE report; output of file command: {report.target.file.type}"
+            f"unrecognized file format from the CAPE report; output of file command: {report.target.file.type}"
         )
 
 
-def extract_os(report: CapeReport) -> Iterator[Tuple[Feature, Address]]:
+def extract_os(report: CapeReport) -> Iterator[tuple[Feature, Address]]:
     # this variable contains the output of the file command
     file_output = report.target.file.type
 
@@ -73,14 +80,14 @@ def extract_os(report: CapeReport) -> Iterator[Tuple[Feature, Address]]:
         else:
             # if the operating system information is missing from the cape report, it's likely a bug
             logger.warning("unrecognized OS: %s", file_output)
-            raise ValueError("unrecognized OS from the CAPE report; output of file command: {file_output}")
+            raise ValueError(f"unrecognized OS from the CAPE report; output of file command: {file_output}")
     else:
         # the sample is shellcode
         logger.debug("unsupported file format, file command output: %s", file_output)
         yield OS(OS_ANY), NO_ADDRESS
 
 
-def extract_features(report: CapeReport) -> Iterator[Tuple[Feature, Address]]:
+def extract_features(report: CapeReport) -> Iterator[tuple[Feature, Address]]:
     for global_handler in GLOBAL_HANDLER:
         for feature, addr in global_handler(report):
             yield feature, addr

capa/features/extractors/cape/helpers.py

@@ -1,17 +1,24 @@
-# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
+# Copyright 2023 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# You may obtain a copy of the License at
-# Unless required by applicable law or agreed to in writing, software distributed under the License
+#
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#     http://www.apache.org/licenses/LICENSE-2.0
-# See the License for the specific language governing permissions and limitations under the License.
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
 
-from typing import Any, Dict, List
+
+from typing import Any
 
 from capa.features.extractors.base_extractor import ProcessHandle
 
 
-def find_process(processes: List[Dict[str, Any]], ph: ProcessHandle) -> Dict[str, Any]:
+def find_process(processes: list[dict[str, Any]], ph: ProcessHandle) -> dict[str, Any]:
     """
     find a specific process identified by a process handler.
 

capa/features/extractors/cape/models.py

@@ -1,15 +1,20 @@
-# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
+# Copyright 2023 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# You may obtain a copy of the License at
-# Unless required by applicable law or agreed to in writing, software distributed under the License
+#
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#     http://www.apache.org/licenses/LICENSE-2.0
-# See the License for the specific language governing permissions and limitations under the License.
+#
-import binascii
+# Unless required by applicable law or agreed to in writing, software
-from typing import Any, Dict, List, Union, Literal, Optional
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from typing import Any, Union, Optional, Annotated, TypeAlias
 
 from pydantic import Field, BaseModel, ConfigDict
-from typing_extensions import Annotated, TypeAlias
 from pydantic.functional_validators import BeforeValidator
 
 
@@ -21,7 +26,7 @@ def validate_hex_int(value):
 
 
 def validate_hex_bytes(value):
-    return binascii.unhexlify(value) if isinstance(value, str) else value
+    return bytes.fromhex(value) if isinstance(value, str) else value
 
 
 HexInt = Annotated[int, BeforeValidator(validate_hex_int)]
@@ -59,45 +64,48 @@ Skip: TypeAlias = Optional[Any]
 # in a field with this type.
 # then we can update the model with the discovered shape.
 TODO: TypeAlias = None
-ListTODO: TypeAlias = List[None]
+ListTODO: TypeAlias = list[None]
 DictTODO: TypeAlias = ExactModel
 
-EmptyDict: TypeAlias = BaseModel
+Emptydict: TypeAlias = BaseModel
-EmptyList: TypeAlias = List[Any]
+EmptyList: TypeAlias = list[Any]
 
 
 class Info(FlexibleModel):
     version: str
 
 
-class ImportedSymbol(ExactModel):
+class ImportedSymbol(FlexibleModel):
     address: HexInt
     name: Optional[str] = None
 
 
-class ImportedDll(ExactModel):
+class ImportedDll(FlexibleModel):
     dll: str
-    imports: List[ImportedSymbol]
+    imports: list[ImportedSymbol]
 
 
-class DirectoryEntry(ExactModel):
+"""
+class DirectoryEntry(FlexibleModel):
     name: str
     virtual_address: HexInt
     size: HexInt
+"""
 
 
-class Section(ExactModel):
+class Section(FlexibleModel):
     name: str
-    raw_address: HexInt
+    # raw_address: HexInt
     virtual_address: HexInt
-    virtual_size: HexInt
+    # virtual_size: HexInt
-    size_of_data: HexInt
+    # size_of_data: HexInt
-    characteristics: str
+    # characteristics: str
-    characteristics_raw: HexInt
+    # characteristics_raw: HexInt
-    entropy: float
+    # entropy: float
 
 
-class Resource(ExactModel):
+"""
+class Resource(FlexibleModel):
     name: str
     language: Optional[str] = None
     sublanguage: str
@@ -135,7 +143,7 @@ class DigitalSigner(FlexibleModel):
     extensions_subjectKeyIdentifier: Optional[str] = None
 
 
-class AuxSigner(ExactModel):
+class AuxSigner(FlexibleModel):
     name: str
     issued_to: str = Field(alias="Issued to")
     issued_by: str = Field(alias="Issued by")
@@ -143,69 +151,70 @@ class AuxSigner(ExactModel):
     sha1_hash: str = Field(alias="SHA1 hash")
 
 
-class Signer(ExactModel):
+class Signer(FlexibleModel):
     aux_sha1: Optional[str] = None
     aux_timestamp: Optional[str] = None
     aux_valid: Optional[bool] = None
     aux_error: Optional[bool] = None
     aux_error_desc: Optional[str] = None
-    aux_signers: Optional[List[AuxSigner]] = None
+    aux_signers: Optional[list[AuxSigner]] = None
 
 
-class Overlay(ExactModel):
+class Overlay(FlexibleModel):
     offset: HexInt
     size: HexInt
 
 
-class KV(ExactModel):
+class KV(FlexibleModel):
     name: str
     value: str
+"""
 
 
-class ExportedSymbol(ExactModel):
+class ExportedSymbol(FlexibleModel):
     address: HexInt
     name: str
-    ordinal: int
+    # ordinal: int
 
 
-class PE(ExactModel):
+class PE(FlexibleModel):
-    peid_signatures: TODO
+    # peid_signatures: TODO
     imagebase: HexInt
-    entrypoint: HexInt
+    # entrypoint: HexInt
-    reported_checksum: HexInt
+    # reported_checksum: HexInt
-    actual_checksum: HexInt
+    # actual_checksum: HexInt
-    osversion: str
+    # osversion: str
-    pdbpath: Optional[str] = None
+    # pdbpath: Optional[str] = None
-    timestamp: str
+    # timestamp: str
 
-    # List[ImportedDll], or Dict[basename(dll), ImportedDll]
+    # list[ImportedDll], or dict[basename(dll), ImportedDll]
-    imports: Union[List[ImportedDll], Dict[str, ImportedDll]]
+    imports: list[ImportedDll] | dict[str, ImportedDll] = Field(default_factory=list)  # type: ignore
-    imported_dll_count: Optional[int] = None
+    # imported_dll_count: Optional[int] = None
-    imphash: str
+    # imphash: str
 
-    exported_dll_name: Optional[str] = None
+    # exported_dll_name: Optional[str] = None
-    exports: List[ExportedSymbol]
+    exports: list[ExportedSymbol] = Field(default_factory=list)
 
-    dirents: List[DirectoryEntry]
+    # dirents: list[DirectoryEntry]
-    sections: List[Section]
+    sections: list[Section] = Field(default_factory=list)
 
-    ep_bytes: Optional[HexBytes] = None
+    # ep_bytes: Optional[HexBytes] = None
 
-    overlay: Optional[Overlay] = None
+    # overlay: Optional[Overlay] = None
-    resources: List[Resource]
+    # resources: list[Resource]
-    versioninfo: List[KV]
+    # versioninfo: list[KV]
 
     # base64 encoded data
-    icon: Optional[str] = None
+    # icon: Optional[str] = None
     # MD5-like hash
-    icon_hash: Optional[str] = None
+    # icon_hash: Optional[str] = None
     # MD5-like hash
-    icon_fuzzy: Optional[str] = None
+    # icon_fuzzy: Optional[str] = None
     # short hex string
-    icon_dhash: Optional[str] = None
+    # icon_dhash: Optional[str] = None
 
-    digital_signers: List[DigitalSigner]
+    # digital_signers: list[DigitalSigner]
-    guest_signers: Signer
+    # guest_signers: Signer
 
 
 # TODO(mr-tz): target.file.dotnet, target.file.extracted_files, target.file.extracted_files_tool,
@@ -213,48 +222,49 @@ class PE(ExactModel):
 # https://github.com/mandiant/capa/issues/1814
 class File(FlexibleModel):
     type: str
-    cape_type_code: Optional[int] = None
+    # cape_type_code: Optional[int] = None
-    cape_type: Optional[str] = None
+    # cape_type: Optional[str] = None
 
-    pid: Optional[Union[int, Literal[""]]] = None
+    # pid: Optional[Union[int, Literal[""]]] = None
-    name: Union[List[str], str]
+    # name: Union[list[str], str]
-    path: str
+    # path: str
-    guest_paths: Union[List[str], str, None]
+    # guest_paths: Union[list[str], str, None]
-    timestamp: Optional[str] = None
+    # timestamp: Optional[str] = None
 
     #
     # hashes
     #
-    crc32: str
+    # crc32: str
     md5: str
     sha1: str
     sha256: str
-    sha512: str
+    # sha512: str
-    sha3_384: Optional[str] = None
+    # sha3_384: Optional[str] = None
-    ssdeep: str
+    # ssdeep: str
     # unsure why this would ever be "False"
-    tlsh: Optional[Union[str, bool]] = None
+    # tlsh: Optional[Union[str, bool]] = None
-    rh_hash: Optional[str] = None
+    # rh_hash: Optional[str] = None
 
     #
     # other metadata, static analysis
     #
-    size: int
+    # size: int
     pe: Optional[PE] = None
-    ep_bytes: Optional[HexBytes] = None
+    # ep_bytes: Optional[HexBytes] = None
-    entrypoint: Optional[int] = None
+    # entrypoint: Optional[int] = None
-    data: Optional[str] = None
+    # data: Optional[str] = None
-    strings: Optional[List[str]] = None
+    # strings: Optional[list[str]] = None
 
     #
     # detections (skip)
     #
-    yara: Skip = None
+    # yara: Skip = None
-    cape_yara: Skip = None
+    # cape_yara: Skip = None
-    clamav: Skip = None
+    # clamav: Skip = None
-    virustotal: Skip = None
+    # virustotal: Skip = None
 
 
+"""
 class ProcessFile(File):
     #
     # like a File, but also has dynamic analysis results
@@ -267,75 +277,82 @@ class ProcessFile(File):
     target_pid: Optional[Union[int, str]] = None
     target_path: Optional[str] = None
     target_process: Optional[str] = None
+"""
 
 
-class Argument(ExactModel):
+class Argument(FlexibleModel):
     name: str
     # unsure why empty list is provided here
     value: Union[HexInt, int, str, EmptyList]
     pretty_value: Optional[str] = None
 
 
-class Call(ExactModel):
+class Call(FlexibleModel):
-    timestamp: str
+    # timestamp: str
     thread_id: int
-    category: str
+    # category: str
 
     api: str
 
-    arguments: List[Argument]
+    arguments: list[Argument]
-    status: bool
+    # status: bool
     return_: HexInt = Field(alias="return")
     pretty_return: Optional[str] = None
 
-    repeated: int
+    # repeated: int
 
     # virtual addresses
-    caller: HexInt
+    # caller: HexInt
-    parentcaller: HexInt
+    # parentcaller: HexInt
 
     # index into calls array
-    id: int
+    # id: int
 
 
-class Process(ExactModel):
+# FlexibleModel to account for extended fields
+# refs: https://github.com/mandiant/capa/issues/2466
+# https://github.com/kevoreilly/CAPEv2/pull/2199
+class Process(FlexibleModel):
     process_id: int
     process_name: str
     parent_id: int
-    module_path: str
+    # module_path: str
-    first_seen: str
+    # first_seen: str
-    calls: List[Call]
+    calls: list[Call]
-    threads: List[int]
+    threads: list[int]
-    environ: Dict[str, str]
+    environ: dict[str, str]
 
 
-class ProcessTree(ExactModel):
+"""
+class ProcessTree(FlexibleModel):
     name: str
     pid: int
     parent_id: int
     module_path: str
-    threads: List[int]
+    threads: list[int]
-    environ: Dict[str, str]
+    environ: dict[str, str]
-    children: List["ProcessTree"]
+    children: list["ProcessTree"]
+"""
 
 
-class Summary(ExactModel):
+class Summary(FlexibleModel):
-    files: List[str]
+    files: list[str]
-    read_files: List[str]
+    # read_files: list[str]
-    write_files: List[str]
+    # write_files: list[str]
-    delete_files: List[str]
+    # delete_files: list[str]
-    keys: List[str]
+    keys: list[str]
-    read_keys: List[str]
+    # read_keys: list[str]
-    write_keys: List[str]
+    # write_keys: list[str]
-    delete_keys: List[str]
+    # delete_keys: list[str]
-    executed_commands: List[str]
+    executed_commands: list[str]
-    resolved_apis: List[str]
+    resolved_apis: list[str]
-    mutexes: List[str]
+    mutexes: list[str]
-    created_services: List[str]
+    created_services: list[str]
-    started_services: List[str]
+    started_services: list[str]
 
 
-class EncryptedBuffer(ExactModel):
+"""
+class EncryptedBuffer(FlexibleModel):
     process_name: str
     pid: int
 
@@ -343,38 +360,41 @@ class EncryptedBuffer(ExactModel):
     buffer: str
     buffer_size: Optional[int] = None
     crypt_key: Optional[Union[HexInt, str]] = None
+"""
 
 
-class Behavior(ExactModel):
+class Behavior(FlexibleModel):
-    summary: Summary
+    summary: Summary | None = None
 
     # list of processes, of threads, of calls
-    processes: List[Process]
+    processes: list[Process]
     # tree of processes
-    processtree: List[ProcessTree]
+    # processtree: list[ProcessTree]
 
-    anomaly: List[str]
+    # anomaly: list[str]
-    encryptedbuffers: List[EncryptedBuffer]
+    # encryptedbuffers: list[EncryptedBuffer]
     # these are small objects that describe atomic events,
     # like file move, registry access.
     # we'll detect the same with our API call analysis.
-    enhanced: Skip = None
+    # enhanced: Skip = None
 
 
-class Target(ExactModel):
+class Target(FlexibleModel):
-    category: str
+    # category: str
     file: File
+    # pe: Optional[PE] = None
+
+
+class Static(FlexibleModel):
     pe: Optional[PE] = None
+    # flare_capa: Skip = None
 
 
-class Static(ExactModel):
+"""
-    pe: Optional[PE] = None
+class Cape(FlexibleModel):
-    flare_capa: Skip = None
+    payloads: list[ProcessFile]
-
-
-class Cape(ExactModel):
-    payloads: List[ProcessFile]
     configs: Skip = None
+"""
 
 
 # flexible because there may be more sorts of analysis
@@ -389,7 +409,7 @@ class CapeReport(FlexibleModel):
     # static analysis results
     #
     static: Optional[Static] = None
-    strings: Optional[List[str]] = None
+    strings: Optional[list[str]] = None
 
     #
     # dynamic analysis results
@@ -397,15 +417,14 @@ class CapeReport(FlexibleModel):
     # post-processed results: process tree, anomalies, etc
     behavior: Behavior
 
-    # post-processed results: payloads and extracted configs
-    CAPE: Optional[Union[Cape, List]] = None
-    dropped: Optional[List[File]] = None
-    procdump: Optional[List[ProcessFile]] = None
-    procmemory: ListTODO
-
     # =========================================================================
     # information we won't use in capa
     #
+    # post-processed results: payloads and extracted configs
+    # CAPE: Optional[Union[Cape, list]] = None
+    # dropped: Optional[list[File]] = None
+    # procdump: Optional[list[ProcessFile]] = None
+    # procmemory: Optional[ListTODO] = None
 
     #
     # NBIs and HBIs
@@ -414,32 +433,32 @@ class CapeReport(FlexibleModel):
     #
     # if we come up with a future use for this, go ahead and re-enable!
     #
-    network: Skip = None
+    # network: Skip = None
-    suricata: Skip = None
+    # suricata: Skip = None
-    curtain: Skip = None
+    # curtain: Skip = None
-    sysmon: Skip = None
+    # sysmon: Skip = None
-    url_analysis: Skip = None
+    # url_analysis: Skip = None
 
     # screenshot hash values
-    deduplicated_shots: Skip = None
+    # deduplicated_shots: Skip = None
     # k-v pairs describing the time it took to run each stage.
-    statistics: Skip = None
+    # statistics: Skip = None
     # k-v pairs of ATT&CK ID to signature name or similar.
-    ttps: Skip = None
+    # ttps: Skip = None
     # debug log messages
-    debug: Skip = None
+    # debug: Skip = None
 
     # various signature matches
     # we could potentially extend capa to use this info one day,
     # though it would be quite sandbox-specific,
     # and more detection-oriented than capability detection.
-    signatures: Skip = None
+    # signatures: Skip = None
-    malfamily_tag: Optional[str] = None
+    # malfamily_tag: Optional[str] = None
-    malscore: float
+    # malscore: float
-    detections: Skip = None
+    # detections: Skip = None
-    detections2pid: Optional[Dict[int, List[str]]] = None
+    # detections2pid: Optional[dict[int, list[str]]] = None
     # AV detections for the sample.
-    virustotal: Skip = None
+    # virustotal: Skip = None
 
     @classmethod
     def from_buf(cls, buf: bytes) -> "CapeReport":

capa/features/extractors/cape/process.py

@@ -1,13 +1,20 @@
-# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
+# Copyright 2023 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# You may obtain a copy of the License at
-# Unless required by applicable law or agreed to in writing, software distributed under the License
+#
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#     http://www.apache.org/licenses/LICENSE-2.0
-# See the License for the specific language governing permissions and limitations under the License.
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 
 import logging
-from typing import List, Tuple, Iterator
+from typing import Iterator
 
 from capa.features.common import String, Feature
 from capa.features.address import Address, ThreadAddress
@@ -22,14 +29,14 @@ def get_threads(ph: ProcessHandle) -> Iterator[ThreadHandle]:
     get the threads associated with a given process
     """
     process: Process = ph.inner
-    threads: List[int] = process.threads
+    threads: list[int] = process.threads
 
     for thread in threads:
         address: ThreadAddress = ThreadAddress(process=ph.address, tid=thread)
         yield ThreadHandle(address=address, inner={})
 
 
-def extract_environ_strings(ph: ProcessHandle) -> Iterator[Tuple[Feature, Address]]:
+def extract_environ_strings(ph: ProcessHandle) -> Iterator[tuple[Feature, Address]]:
     """
     extract strings from a process' provided environment variables.
     """
@@ -39,7 +46,7 @@ def extract_environ_strings(ph: ProcessHandle) -> Iterator[Tuple[Feature, Addres
         yield String(value), ph.address
 
 
-def extract_features(ph: ProcessHandle) -> Iterator[Tuple[Feature, Address]]:
+def extract_features(ph: ProcessHandle) -> Iterator[tuple[Feature, Address]]:
     for handler in PROCESS_HANDLERS:
         for feature, addr in handler(ph):
             yield feature, addr

capa/features/extractors/cape/thread.py

@@ -1,10 +1,17 @@
-# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
+# Copyright 2023 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# You may obtain a copy of the License at
-# Unless required by applicable law or agreed to in writing, software distributed under the License
+#
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#     http://www.apache.org/licenses/LICENSE-2.0
-# See the License for the specific language governing permissions and limitations under the License.
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 
 import logging
 from typing import Iterator

capa/features/extractors/common.py

@@ -1,16 +1,23 @@
-# Copyright (C) 2021 Mandiant, Inc. All Rights Reserved.
+# Copyright 2021 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# You may obtain a copy of the License at
-# Unless required by applicable law or agreed to in writing, software distributed under the License
+#
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#     http://www.apache.org/licenses/LICENSE-2.0
-# See the License for the specific language governing permissions and limitations under the License.
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 import io
 import re
 import logging
 import binascii
 import contextlib
-from typing import Tuple, Iterator
+from typing import Iterator
 
 import pefile
 
@@ -45,7 +52,7 @@ MATCH_RESULT = b'{"meta":'
 MATCH_JSON_OBJECT = b'{"'
 
 
-def extract_file_strings(buf: bytes, **kwargs) -> Iterator[Tuple[String, Address]]:
+def extract_file_strings(buf: bytes, **kwargs) -> Iterator[tuple[String, Address]]:
     """
     extract ASCII and UTF-16 LE strings from file
     """
@@ -56,7 +63,7 @@ def extract_file_strings(buf: bytes, **kwargs) -> Iterator[Tuple[String, Address
         yield String(s.s), FileOffsetAddress(s.offset)
 
 
-def extract_format(buf: bytes) -> Iterator[Tuple[Feature, Address]]:
+def extract_format(buf: bytes) -> Iterator[tuple[Feature, Address]]:
     if buf.startswith(MATCH_PE):
         yield Format(FORMAT_PE), NO_ADDRESS
     elif buf.startswith(MATCH_ELF):
@@ -75,11 +82,11 @@ def extract_format(buf: bytes) -> Iterator[Tuple[Feature, Address]]:
         #  1. handling a file format (e.g. macho)
         #
         # for (1), this logic will need to be updated as the format is implemented.
-        logger.debug("unsupported file format: %s", binascii.hexlify(buf[:4]).decode("ascii"))
+        logger.debug("unknown file format: %s", buf[:4].hex())
         return
 
 
-def extract_arch(buf) -> Iterator[Tuple[Feature, Address]]:
+def extract_arch(buf) -> Iterator[tuple[Feature, Address]]:
     if buf.startswith(MATCH_PE):
         yield from capa.features.extractors.pefile.extract_file_arch(pe=pefile.PE(data=buf))
 
@@ -111,7 +118,7 @@ def extract_arch(buf) -> Iterator[Tuple[Feature, Address]]:
         return
 
 
-def extract_os(buf, os=OS_AUTO) -> Iterator[Tuple[Feature, Address]]:
+def extract_os(buf, os=OS_AUTO) -> Iterator[tuple[Feature, Address]]:
     if os != OS_AUTO:
         yield OS(os), NO_ADDRESS
 

capa/features/extractors/dnfile/extractor.py

@@ -1,14 +1,21 @@
-# Copyright (C) 2022 Mandiant, Inc. All Rights Reserved.
+# Copyright 2022 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# You may obtain a copy of the License at
-# Unless required by applicable law or agreed to in writing, software distributed under the License
+#
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#     http://www.apache.org/licenses/LICENSE-2.0
-# See the License for the specific language governing permissions and limitations under the License.
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 
 from __future__ import annotations
 
-from typing import Dict, List, Tuple, Union, Iterator, Optional
+from typing import Union, Iterator, Optional
 from pathlib import Path
 
 import dnfile
@@ -41,11 +48,11 @@ from capa.features.extractors.dnfile.helpers import (
 
 class DnFileFeatureExtractorCache:
     def __init__(self, pe: dnfile.dnPE):
-        self.imports: Dict[int, Union[DnType, DnUnmanagedMethod]] = {}
+        self.imports: dict[int, Union[DnType, DnUnmanagedMethod]] = {}
-        self.native_imports: Dict[int, Union[DnType, DnUnmanagedMethod]] = {}
+        self.native_imports: dict[int, Union[DnType, DnUnmanagedMethod]] = {}
-        self.methods: Dict[int, Union[DnType, DnUnmanagedMethod]] = {}
+        self.methods: dict[int, Union[DnType, DnUnmanagedMethod]] = {}
-        self.fields: Dict[int, Union[DnType, DnUnmanagedMethod]] = {}
+        self.fields: dict[int, Union[DnType, DnUnmanagedMethod]] = {}
-        self.types: Dict[int, Union[DnType, DnUnmanagedMethod]] = {}
+        self.types: dict[int, Union[DnType, DnUnmanagedMethod]] = {}
 
         for import_ in get_dotnet_managed_imports(pe):
             self.imports[import_.token] = import_
@@ -84,7 +91,7 @@ class DnfileFeatureExtractor(StaticFeatureExtractor):
         self.token_cache: DnFileFeatureExtractorCache = DnFileFeatureExtractorCache(self.pe)
 
         # pre-compute these because we'll yield them at *every* scope.
-        self.global_features: List[Tuple[Feature, Address]] = []
+        self.global_features: list[tuple[Feature, Address]] = []
         self.global_features.extend(capa.features.extractors.dotnetfile.extract_file_format())
         self.global_features.extend(capa.features.extractors.dotnetfile.extract_file_os(pe=self.pe))
         self.global_features.extend(capa.features.extractors.dotnetfile.extract_file_arch(pe=self.pe))
@@ -100,7 +107,7 @@ class DnfileFeatureExtractor(StaticFeatureExtractor):
 
     def get_functions(self) -> Iterator[FunctionHandle]:
         # create a method lookup table
-        methods: Dict[Address, FunctionHandle] = {}
+        methods: dict[Address, FunctionHandle] = {}
         for token, method in get_dotnet_managed_method_bodies(self.pe):
             fh: FunctionHandle = FunctionHandle(
                 address=DNTokenAddress(token),
@@ -136,7 +143,7 @@ class DnfileFeatureExtractor(StaticFeatureExtractor):
 
         yield from methods.values()
 
-    def extract_function_features(self, fh) -> Iterator[Tuple[Feature, Address]]:
+    def extract_function_features(self, fh) -> Iterator[tuple[Feature, Address]]:
         yield from capa.features.extractors.dnfile.function.extract_features(fh)
 
     def get_basic_blocks(self, f) -> Iterator[BBHandle]:
@@ -157,5 +164,5 @@ class DnfileFeatureExtractor(StaticFeatureExtractor):
                 inner=insn,
             )
 
-    def extract_insn_features(self, fh, bbh, ih) -> Iterator[Tuple[Feature, Address]]:
+    def extract_insn_features(self, fh, bbh, ih) -> Iterator[tuple[Feature, Address]]:
         yield from capa.features.extractors.dnfile.insn.extract_features(fh, bbh, ih)

capa/features/extractors/dnfile/file.py

@@ -1,14 +1,21 @@
-# Copyright (C) 2022 Mandiant, Inc. All Rights Reserved.
+# Copyright 2022 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# You may obtain a copy of the License at
-# Unless required by applicable law or agreed to in writing, software distributed under the License
+#
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#     http://www.apache.org/licenses/LICENSE-2.0
-# See the License for the specific language governing permissions and limitations under the License.
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 
 from __future__ import annotations
 
-from typing import Tuple, Iterator
+from typing import Iterator
 
 import dnfile
 
@@ -18,35 +25,35 @@ from capa.features.common import Class, Format, String, Feature, Namespace, Char
 from capa.features.address import Address
 
 
-def extract_file_import_names(pe: dnfile.dnPE) -> Iterator[Tuple[Import, Address]]:
+def extract_file_import_names(pe: dnfile.dnPE) -> Iterator[tuple[Import, Address]]:
     yield from capa.features.extractors.dotnetfile.extract_file_import_names(pe=pe)
 
 
-def extract_file_format(pe: dnfile.dnPE) -> Iterator[Tuple[Format, Address]]:
+def extract_file_format(pe: dnfile.dnPE) -> Iterator[tuple[Format, Address]]:
     yield from capa.features.extractors.dotnetfile.extract_file_format(pe=pe)
 
 
-def extract_file_function_names(pe: dnfile.dnPE) -> Iterator[Tuple[FunctionName, Address]]:
+def extract_file_function_names(pe: dnfile.dnPE) -> Iterator[tuple[FunctionName, Address]]:
     yield from capa.features.extractors.dotnetfile.extract_file_function_names(pe=pe)
 
 
-def extract_file_strings(pe: dnfile.dnPE) -> Iterator[Tuple[String, Address]]:
+def extract_file_strings(pe: dnfile.dnPE) -> Iterator[tuple[String, Address]]:
     yield from capa.features.extractors.dotnetfile.extract_file_strings(pe=pe)
 
 
-def extract_file_mixed_mode_characteristic_features(pe: dnfile.dnPE) -> Iterator[Tuple[Characteristic, Address]]:
+def extract_file_mixed_mode_characteristic_features(pe: dnfile.dnPE) -> Iterator[tuple[Characteristic, Address]]:
     yield from capa.features.extractors.dotnetfile.extract_file_mixed_mode_characteristic_features(pe=pe)
 
 
-def extract_file_namespace_features(pe: dnfile.dnPE) -> Iterator[Tuple[Namespace, Address]]:
+def extract_file_namespace_features(pe: dnfile.dnPE) -> Iterator[tuple[Namespace, Address]]:
     yield from capa.features.extractors.dotnetfile.extract_file_namespace_features(pe=pe)
 
 
-def extract_file_class_features(pe: dnfile.dnPE) -> Iterator[Tuple[Class, Address]]:
+def extract_file_class_features(pe: dnfile.dnPE) -> Iterator[tuple[Class, Address]]:
     yield from capa.features.extractors.dotnetfile.extract_file_class_features(pe=pe)
 
 
-def extract_features(pe: dnfile.dnPE) -> Iterator[Tuple[Feature, Address]]:
+def extract_features(pe: dnfile.dnPE) -> Iterator[tuple[Feature, Address]]:
     for file_handler in FILE_HANDLERS:
         for feature, address in file_handler(pe):
             yield feature, address

capa/features/extractors/dnfile/function.py

@@ -1,15 +1,22 @@
-# Copyright (C) 2022 Mandiant, Inc. All Rights Reserved.
+# Copyright 2022 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# You may obtain a copy of the License at
-# Unless required by applicable law or agreed to in writing, software distributed under the License
+#
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#     http://www.apache.org/licenses/LICENSE-2.0
-# See the License for the specific language governing permissions and limitations under the License.
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 
 from __future__ import annotations
 
 import logging
-from typing import Tuple, Iterator
+from typing import Iterator
 
 from capa.features.common import Feature, Characteristic
 from capa.features.address import Address
@@ -18,30 +25,30 @@ from capa.features.extractors.base_extractor import FunctionHandle
 logger = logging.getLogger(__name__)
 
 
-def extract_function_calls_to(fh: FunctionHandle) -> Iterator[Tuple[Characteristic, Address]]:
+def extract_function_calls_to(fh: FunctionHandle) -> Iterator[tuple[Characteristic, Address]]:
     """extract callers to a function"""
     for dest in fh.ctx["calls_to"]:
         yield Characteristic("calls to"), dest
 
 
-def extract_function_calls_from(fh: FunctionHandle) -> Iterator[Tuple[Characteristic, Address]]:
+def extract_function_calls_from(fh: FunctionHandle) -> Iterator[tuple[Characteristic, Address]]:
     """extract callers from a function"""
     for src in fh.ctx["calls_from"]:
         yield Characteristic("calls from"), src
 
 
-def extract_recursive_call(fh: FunctionHandle) -> Iterator[Tuple[Characteristic, Address]]:
+def extract_recursive_call(fh: FunctionHandle) -> Iterator[tuple[Characteristic, Address]]:
     """extract recursive function call"""
     if fh.address in fh.ctx["calls_to"]:
         yield Characteristic("recursive call"), fh.address
 
 
-def extract_function_loop(fh: FunctionHandle) -> Iterator[Tuple[Characteristic, Address]]:
+def extract_function_loop(fh: FunctionHandle) -> Iterator[tuple[Characteristic, Address]]:
     """extract loop indicators from a function"""
     raise NotImplementedError()
 
 
-def extract_features(fh: FunctionHandle) -> Iterator[Tuple[Feature, Address]]:
+def extract_features(fh: FunctionHandle) -> Iterator[tuple[Feature, Address]]:
     for func_handler in FUNCTION_HANDLERS:
         for feature, addr in func_handler(fh):
             yield feature, addr

capa/features/extractors/dnfile/helpers.py

@@ -1,15 +1,22 @@
-# Copyright (C) 2022 Mandiant, Inc. All Rights Reserved.
+# Copyright 2022 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# You may obtain a copy of the License at
-# Unless required by applicable law or agreed to in writing, software distributed under the License
+#
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#     http://www.apache.org/licenses/LICENSE-2.0
-# See the License for the specific language governing permissions and limitations under the License.
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 
 from __future__ import annotations
 
 import logging
-from typing import Dict, Tuple, Union, Iterator, Optional
+from typing import Union, Iterator, Optional
 
 import dnfile
 from dncil.cil.body import CilMethodBody
@@ -144,7 +151,7 @@ def get_dotnet_managed_imports(pe: dnfile.dnPE) -> Iterator[DnType]:
         )
 
 
-def get_dotnet_methoddef_property_accessors(pe: dnfile.dnPE) -> Iterator[Tuple[int, str]]:
+def get_dotnet_methoddef_property_accessors(pe: dnfile.dnPE) -> Iterator[tuple[int, str]]:
     """get MethodDef methods used to access properties
 
     see https://www.ntcore.com/files/dotnetformat.htm
@@ -194,7 +201,7 @@ def get_dotnet_managed_methods(pe: dnfile.dnPE) -> Iterator[DnType]:
     """
     nested_class_table = get_dotnet_nested_class_table_index(pe)
 
-    accessor_map: Dict[int, str] = {}
+    accessor_map: dict[int, str] = {}
     for methoddef, methoddef_access in get_dotnet_methoddef_property_accessors(pe):
         accessor_map[methoddef] = methoddef_access
 
@@ -252,7 +259,7 @@ def get_dotnet_fields(pe: dnfile.dnPE) -> Iterator[DnType]:
             yield DnType(token, typedefname, namespace=typedefnamespace, member=field.row.Name)
 
 
-def get_dotnet_managed_method_bodies(pe: dnfile.dnPE) -> Iterator[Tuple[int, CilMethodBody]]:
+def get_dotnet_managed_method_bodies(pe: dnfile.dnPE) -> Iterator[tuple[int, CilMethodBody]]:
     """get managed methods from MethodDef table"""
     for rid, method_def in iter_dotnet_table(pe, dnfile.mdtable.MethodDef.number):
         assert isinstance(method_def, dnfile.mdtable.MethodDefRow)
@@ -332,7 +339,7 @@ def get_dotnet_table_row(pe: dnfile.dnPE, table_index: int, row_index: int) -> O
 
 def resolve_nested_typedef_name(
     nested_class_table: dict, index: int, typedef: dnfile.mdtable.TypeDefRow, pe: dnfile.dnPE
-) -> Tuple[str, Tuple[str, ...]]:
+) -> tuple[str, tuple[str, ...]]:
     """Resolves all nested TypeDef class names. Returns the namespace as a str and the nested TypeRef name as a tuple"""
 
     if index in nested_class_table:
@@ -368,7 +375,7 @@ def resolve_nested_typedef_name(
 
 def resolve_nested_typeref_name(
     index: int, typeref: dnfile.mdtable.TypeRefRow, pe: dnfile.dnPE
-) -> Tuple[str, Tuple[str, ...]]:
+) -> tuple[str, tuple[str, ...]]:
     """Resolves all nested TypeRef class names. Returns the namespace as a str and the nested TypeRef name as a tuple"""
     # If the ResolutionScope decodes to a typeRef type then it is nested
     if isinstance(typeref.ResolutionScope.table, dnfile.mdtable.TypeRef):
@@ -398,7 +405,7 @@ def resolve_nested_typeref_name(
         return str(typeref.TypeNamespace), (str(typeref.TypeName),)
 
 
-def get_dotnet_nested_class_table_index(pe: dnfile.dnPE) -> Dict[int, int]:
+def get_dotnet_nested_class_table_index(pe: dnfile.dnPE) -> dict[int, int]:
     """Build index for EnclosingClass based off the NestedClass row index in the nestedclass table"""
     nested_class_table = {}
 
@@ -442,7 +449,7 @@ def is_dotnet_mixed_mode(pe: dnfile.dnPE) -> bool:
     return not bool(pe.net.Flags.CLR_ILONLY)
 
 
-def iter_dotnet_table(pe: dnfile.dnPE, table_index: int) -> Iterator[Tuple[int, dnfile.base.MDTableRow]]:
+def iter_dotnet_table(pe: dnfile.dnPE, table_index: int) -> Iterator[tuple[int, dnfile.base.MDTableRow]]:
     assert pe.net is not None
     assert pe.net.mdtables is not None
 

capa/features/extractors/dnfile/insn.py

@@ -1,15 +1,22 @@
-# Copyright (C) 2022 Mandiant, Inc. All Rights Reserved.
+# Copyright 2022 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# You may obtain a copy of the License at
-# Unless required by applicable law or agreed to in writing, software distributed under the License
+#
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#     http://www.apache.org/licenses/LICENSE-2.0
-# See the License for the specific language governing permissions and limitations under the License.
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 
 from __future__ import annotations
 
 import logging
-from typing import TYPE_CHECKING, Tuple, Union, Iterator, Optional
+from typing import TYPE_CHECKING, Union, Iterator, Optional
 
 if TYPE_CHECKING:
     from capa.features.extractors.dnfile.extractor import DnFileFeatureExtractorCache
@@ -61,7 +68,7 @@ def get_callee(
     return callee
 
 
-def extract_insn_api_features(fh: FunctionHandle, bh, ih: InsnHandle) -> Iterator[Tuple[Feature, Address]]:
+def extract_insn_api_features(fh: FunctionHandle, bh, ih: InsnHandle) -> Iterator[tuple[Feature, Address]]:
     """parse instruction API features"""
     if ih.inner.opcode not in (
         OpCodes.Call,
@@ -83,7 +90,7 @@ def extract_insn_api_features(fh: FunctionHandle, bh, ih: InsnHandle) -> Iterato
             yield API(name), ih.address
 
 
-def extract_insn_property_features(fh: FunctionHandle, bh, ih: InsnHandle) -> Iterator[Tuple[Feature, Address]]:
+def extract_insn_property_features(fh: FunctionHandle, bh, ih: InsnHandle) -> Iterator[tuple[Feature, Address]]:
     """parse instruction property features"""
     name: Optional[str] = None
     access: Optional[str] = None
@@ -118,7 +125,7 @@ def extract_insn_property_features(fh: FunctionHandle, bh, ih: InsnHandle) -> It
 
 def extract_insn_namespace_class_features(
     fh: FunctionHandle, bh, ih: InsnHandle
-) -> Iterator[Tuple[Union[Namespace, Class], Address]]:
+) -> Iterator[tuple[Union[Namespace, Class], Address]]:
     """parse instruction namespace and class features"""
     type_: Optional[Union[DnType, DnUnmanagedMethod]] = None
 
@@ -173,13 +180,13 @@ def extract_insn_namespace_class_features(
             yield Namespace(type_.namespace), ih.address
 
 
-def extract_insn_number_features(fh, bh, ih: InsnHandle) -> Iterator[Tuple[Feature, Address]]:
+def extract_insn_number_features(fh, bh, ih: InsnHandle) -> Iterator[tuple[Feature, Address]]:
     """parse instruction number features"""
     if ih.inner.is_ldc():
         yield Number(ih.inner.get_ldc()), ih.address
 
 
-def extract_insn_string_features(fh: FunctionHandle, bh, ih: InsnHandle) -> Iterator[Tuple[Feature, Address]]:
+def extract_insn_string_features(fh: FunctionHandle, bh, ih: InsnHandle) -> Iterator[tuple[Feature, Address]]:
     """parse instruction string features"""
     if not ih.inner.is_ldstr():
         return
@@ -197,7 +204,7 @@ def extract_insn_string_features(fh: FunctionHandle, bh, ih: InsnHandle) -> Iter
 
 def extract_unmanaged_call_characteristic_features(
     fh: FunctionHandle, bb: BBHandle, ih: InsnHandle
-) -> Iterator[Tuple[Characteristic, Address]]:
+) -> Iterator[tuple[Characteristic, Address]]:
     if ih.inner.opcode not in (OpCodes.Call, OpCodes.Callvirt, OpCodes.Jmp):
         return
 
@@ -209,7 +216,7 @@ def extract_unmanaged_call_characteristic_features(
         yield Characteristic("unmanaged call"), ih.address
 
 
-def extract_features(fh: FunctionHandle, bbh: BBHandle, ih: InsnHandle) -> Iterator[Tuple[Feature, Address]]:
+def extract_features(fh: FunctionHandle, bbh: BBHandle, ih: InsnHandle) -> Iterator[tuple[Feature, Address]]:
     """extract instruction features"""
     for inst_handler in INSTRUCTION_HANDLERS:
         for feature, addr in inst_handler(fh, bbh, ih):

capa/features/extractors/dnfile/types.py

@@ -1,22 +1,29 @@
-# Copyright (C) 2022 Mandiant, Inc. All Rights Reserved.
+# Copyright 2022 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# You may obtain a copy of the License at
-# Unless required by applicable law or agreed to in writing, software distributed under the License
+#
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#     http://www.apache.org/licenses/LICENSE-2.0
-# See the License for the specific language governing permissions and limitations under the License.
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
 
-from typing import Tuple, Optional
+
+from typing import Optional
 
 
 class DnType:
     def __init__(
-        self, token: int, class_: Tuple[str, ...], namespace: str = "", member: str = "", access: Optional[str] = None
+        self, token: int, class_: tuple[str, ...], namespace: str = "", member: str = "", access: Optional[str] = None
     ):
         self.token: int = token
         self.access: Optional[str] = access
         self.namespace: str = namespace
-        self.class_: Tuple[str, ...] = class_
+        self.class_: tuple[str, ...] = class_
 
         if member == ".ctor":
             member = "ctor"
@@ -44,7 +51,7 @@ class DnType:
         return str(self)
 
     @staticmethod
-    def format_name(class_: Tuple[str, ...], namespace: str = "", member: str = ""):
+    def format_name(class_: tuple[str, ...], namespace: str = "", member: str = ""):
         if len(class_) > 1:
             class_str = "/".join(class_)  # Concat items in tuple, separated by a "/"
         else:

capa/features/extractors/dotnetfile.py

@@ -1,12 +1,19 @@
-# Copyright (C) 2022 Mandiant, Inc. All Rights Reserved.
+# Copyright 2022 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# You may obtain a copy of the License at
-# Unless required by applicable law or agreed to in writing, software distributed under the License
+#
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#     http://www.apache.org/licenses/LICENSE-2.0
-# See the License for the specific language governing permissions and limitations under the License.
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 import logging
-from typing import Tuple, Iterator
+from typing import Iterator
 from pathlib import Path
 
 import dnfile
@@ -48,12 +55,12 @@ from capa.features.extractors.dnfile.helpers import (
 logger = logging.getLogger(__name__)
 
 
-def extract_file_format(**kwargs) -> Iterator[Tuple[Format, Address]]:
+def extract_file_format(**kwargs) -> Iterator[tuple[Format, Address]]:
     yield Format(FORMAT_DOTNET), NO_ADDRESS
     yield Format(FORMAT_PE), NO_ADDRESS
 
 
-def extract_file_import_names(pe: dnfile.dnPE, **kwargs) -> Iterator[Tuple[Import, Address]]:
+def extract_file_import_names(pe: dnfile.dnPE, **kwargs) -> Iterator[tuple[Import, Address]]:
     for method in get_dotnet_managed_imports(pe):
         # like System.IO.File::OpenRead
         yield Import(str(method)), DNTokenAddress(method.token)
@@ -64,12 +71,12 @@ def extract_file_import_names(pe: dnfile.dnPE, **kwargs) -> Iterator[Tuple[Impor
             yield Import(name), DNTokenAddress(imp.token)
 
 
-def extract_file_function_names(pe: dnfile.dnPE, **kwargs) -> Iterator[Tuple[FunctionName, Address]]:
+def extract_file_function_names(pe: dnfile.dnPE, **kwargs) -> Iterator[tuple[FunctionName, Address]]:
     for method in get_dotnet_managed_methods(pe):
         yield FunctionName(str(method)), DNTokenAddress(method.token)
 
 
-def extract_file_namespace_features(pe: dnfile.dnPE, **kwargs) -> Iterator[Tuple[Namespace, Address]]:
+def extract_file_namespace_features(pe: dnfile.dnPE, **kwargs) -> Iterator[tuple[Namespace, Address]]:
     """emit namespace features from TypeRef and TypeDef tables"""
 
     # namespaces may be referenced multiple times, so we need to filter
@@ -93,7 +100,7 @@ def extract_file_namespace_features(pe: dnfile.dnPE, **kwargs) -> Iterator[Tuple
         yield Namespace(namespace), NO_ADDRESS
 
 
-def extract_file_class_features(pe: dnfile.dnPE, **kwargs) -> Iterator[Tuple[Class, Address]]:
+def extract_file_class_features(pe: dnfile.dnPE, **kwargs) -> Iterator[tuple[Class, Address]]:
     """emit class features from TypeRef and TypeDef tables"""
     nested_class_table = get_dotnet_nested_class_table_index(pe)
 
@@ -116,11 +123,11 @@ def extract_file_class_features(pe: dnfile.dnPE, **kwargs) -> Iterator[Tuple[Cla
         yield Class(DnType.format_name(typerefname, namespace=typerefnamespace)), DNTokenAddress(token)
 
 
-def extract_file_os(**kwargs) -> Iterator[Tuple[OS, Address]]:
+def extract_file_os(**kwargs) -> Iterator[tuple[OS, Address]]:
     yield OS(OS_ANY), NO_ADDRESS
 
 
-def extract_file_arch(pe: dnfile.dnPE, **kwargs) -> Iterator[Tuple[Arch, Address]]:
+def extract_file_arch(pe: dnfile.dnPE, **kwargs) -> Iterator[tuple[Arch, Address]]:
     # to distinguish in more detail, see https://stackoverflow.com/a/23614024/10548020
     # .NET 4.5 added option: any CPU, 32-bit preferred
     assert pe.net is not None
@@ -134,18 +141,18 @@ def extract_file_arch(pe: dnfile.dnPE, **kwargs) -> Iterator[Tuple[Arch, Address
         yield Arch(ARCH_ANY), NO_ADDRESS
 
 
-def extract_file_strings(pe: dnfile.dnPE, **kwargs) -> Iterator[Tuple[String, Address]]:
+def extract_file_strings(pe: dnfile.dnPE, **kwargs) -> Iterator[tuple[String, Address]]:
     yield from capa.features.extractors.common.extract_file_strings(pe.__data__)
 
 
 def extract_file_mixed_mode_characteristic_features(
     pe: dnfile.dnPE, **kwargs
-) -> Iterator[Tuple[Characteristic, Address]]:
+) -> Iterator[tuple[Characteristic, Address]]:
     if is_dotnet_mixed_mode(pe):
         yield Characteristic("mixed mode"), NO_ADDRESS
 
 
-def extract_file_features(pe: dnfile.dnPE) -> Iterator[Tuple[Feature, Address]]:
+def extract_file_features(pe: dnfile.dnPE) -> Iterator[tuple[Feature, Address]]:
     for file_handler in FILE_HANDLERS:
         for feature, addr in file_handler(pe=pe):  # type: ignore
             yield feature, addr
@@ -162,7 +169,7 @@ FILE_HANDLERS = (
 )
 
 
-def extract_global_features(pe: dnfile.dnPE) -> Iterator[Tuple[Feature, Address]]:
+def extract_global_features(pe: dnfile.dnPE) -> Iterator[tuple[Feature, Address]]:
     for handler in GLOBAL_HANDLERS:
         for feature, va in handler(pe=pe):  # type: ignore
             yield feature, va
@@ -204,7 +211,7 @@ class DotnetFileFeatureExtractor(StaticFeatureExtractor):
     def is_mixed_mode(self) -> bool:
         return is_dotnet_mixed_mode(self.pe)
 
-    def get_runtime_version(self) -> Tuple[int, int]:
+    def get_runtime_version(self) -> tuple[int, int]:
         assert self.pe.net is not None
         assert self.pe.net.struct is not None
         assert self.pe.net.struct.MajorRuntimeVersion is not None

capa/features/extractors/drakvuf/call.py

@@ -0,0 +1,65 @@
+# Copyright 2024 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+import logging
+from typing import Iterator
+
+import capa.features.extractors.helpers
+from capa.features.insn import API, Number
+from capa.features.common import String, Feature
+from capa.features.address import Address
+from capa.features.extractors.base_extractor import CallHandle, ThreadHandle, ProcessHandle
+from capa.features.extractors.drakvuf.models import Call
+
+logger = logging.getLogger(__name__)
+
+
+def extract_call_features(ph: ProcessHandle, th: ThreadHandle, ch: CallHandle) -> Iterator[tuple[Feature, Address]]:
+    """
+    This method extracts the given call's features (such as API name and arguments),
+    and returns them as API, Number, and String features.
+
+    args:
+      ph: process handle (for defining the extraction scope)
+      th: thread handle (for defining the extraction scope)
+      ch: call handle (for defining the extraction scope)
+
+    yields:
+      Feature, address; where Feature is either: API, Number, or String.
+    """
+    call: Call = ch.inner
+
+    # list similar to disassembly: arguments right-to-left, call
+    for arg_value in reversed(call.arguments.values()):
+        try:
+            yield Number(int(arg_value, 0)), ch.address
+        except ValueError:
+            # DRAKVUF automatically resolves the contents of memory addresses, (e.g. Arg1="0xc6f217efe0:\"ntdll.dll\"").
+            # For those cases we yield the entire string as it, since yielding the address only would
+            # likely not provide any matches, and yielding just the memory contentswould probably be misleading,
+            # but yielding the entire string would be helpful for an analyst looking at the verbose output
+            yield String(arg_value), ch.address
+
+    for name in capa.features.extractors.helpers.generate_symbols("", call.name):
+        yield API(name), ch.address
+
+
+def extract_features(ph: ProcessHandle, th: ThreadHandle, ch: CallHandle) -> Iterator[tuple[Feature, Address]]:
+    for handler in CALL_HANDLERS:
+        for feature, addr in handler(ph, th, ch):
+            yield feature, addr
+
+
+CALL_HANDLERS = (extract_call_features,)

capa/features/extractors/drakvuf/extractor.py

@@ -0,0 +1,99 @@
+# Copyright 2024 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+import logging
+from typing import Union, Iterator
+
+import capa.features.extractors.drakvuf.call
+import capa.features.extractors.drakvuf.file
+import capa.features.extractors.drakvuf.thread
+import capa.features.extractors.drakvuf.global_
+import capa.features.extractors.drakvuf.process
+from capa.features.common import Feature
+from capa.features.address import NO_ADDRESS, Address, ThreadAddress, ProcessAddress, AbsoluteVirtualAddress, _NoAddress
+from capa.features.extractors.base_extractor import (
+    CallHandle,
+    SampleHashes,
+    ThreadHandle,
+    ProcessHandle,
+    DynamicFeatureExtractor,
+)
+from capa.features.extractors.drakvuf.models import Call, DrakvufReport
+from capa.features.extractors.drakvuf.helpers import index_calls
+
+logger = logging.getLogger(__name__)
+
+
+class DrakvufExtractor(DynamicFeatureExtractor):
+    def __init__(self, report: DrakvufReport):
+        super().__init__(
+            # DRAKVUF currently does not yield hash information about the sample in its output
+            hashes=SampleHashes(md5="", sha1="", sha256="")
+        )
+
+        self.report: DrakvufReport = report
+
+        # sort the api calls to prevent going through the entire list each time
+        self.sorted_calls: dict[ProcessAddress, dict[ThreadAddress, list[Call]]] = index_calls(report)
+
+        # pre-compute these because we'll yield them at *every* scope.
+        self.global_features = list(capa.features.extractors.drakvuf.global_.extract_features(self.report))
+
+    def get_base_address(self) -> Union[AbsoluteVirtualAddress, _NoAddress, None]:
+        # DRAKVUF currently does not yield information about the PE's address
+        return NO_ADDRESS
+
+    def extract_global_features(self) -> Iterator[tuple[Feature, Address]]:
+        yield from self.global_features
+
+    def extract_file_features(self) -> Iterator[tuple[Feature, Address]]:
+        yield from capa.features.extractors.drakvuf.file.extract_features(self.report)
+
+    def get_processes(self) -> Iterator[ProcessHandle]:
+        yield from capa.features.extractors.drakvuf.file.get_processes(self.sorted_calls)
+
+    def extract_process_features(self, ph: ProcessHandle) -> Iterator[tuple[Feature, Address]]:
+        yield from capa.features.extractors.drakvuf.process.extract_features(ph)
+
+    def get_process_name(self, ph: ProcessHandle) -> str:
+        return ph.inner["process_name"]
+
+    def get_threads(self, ph: ProcessHandle) -> Iterator[ThreadHandle]:
+        yield from capa.features.extractors.drakvuf.process.get_threads(self.sorted_calls, ph)
+
+    def extract_thread_features(self, ph: ProcessHandle, th: ThreadHandle) -> Iterator[tuple[Feature, Address]]:
+        yield from []
+
+    def get_calls(self, ph: ProcessHandle, th: ThreadHandle) -> Iterator[CallHandle]:
+        yield from capa.features.extractors.drakvuf.thread.get_calls(self.sorted_calls, ph, th)
+
+    def get_call_name(self, ph: ProcessHandle, th: ThreadHandle, ch: CallHandle) -> str:
+        call: Call = ch.inner
+        call_name = "{}({}){}".format(
+            call.name,
+            ", ".join(f"{arg_name}={arg_value}" for arg_name, arg_value in call.arguments.items()),
+            (f" -> {getattr(call, 'return_value', '')}"),  # SysCalls don't have a return value, while WinApi calls do
+        )
+        return call_name
+
+    def extract_call_features(
+        self, ph: ProcessHandle, th: ThreadHandle, ch: CallHandle
+    ) -> Iterator[tuple[Feature, Address]]:
+        yield from capa.features.extractors.drakvuf.call.extract_features(ph, th, ch)
+
+    @classmethod
+    def from_report(cls, report: Iterator[dict]) -> "DrakvufExtractor":
+        dr = DrakvufReport.from_raw_report(report)
+        return DrakvufExtractor(report=dr)

capa/features/extractors/drakvuf/file.py

@@ -0,0 +1,63 @@
+# Copyright 2024 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+import logging
+from typing import Iterator
+
+from capa.features.file import Import
+from capa.features.common import Feature
+from capa.features.address import Address, ThreadAddress, ProcessAddress, AbsoluteVirtualAddress
+from capa.features.extractors.helpers import generate_symbols
+from capa.features.extractors.base_extractor import ProcessHandle
+from capa.features.extractors.drakvuf.models import Call, DrakvufReport
+
+logger = logging.getLogger(__name__)
+
+
+def get_processes(calls: dict[ProcessAddress, dict[ThreadAddress, list[Call]]]) -> Iterator[ProcessHandle]:
+    """
+    Get all the created processes for a sample.
+    """
+    for proc_addr, calls_per_thread in calls.items():
+        sample_call = next(iter(calls_per_thread.values()))[0]  # get process name
+        yield ProcessHandle(proc_addr, inner={"process_name": sample_call.process_name})
+
+
+def extract_import_names(report: DrakvufReport) -> Iterator[tuple[Feature, Address]]:
+    """
+    Extract imported function names.
+    """
+    if report.loaded_dlls is None:
+        return
+    dlls = report.loaded_dlls
+
+    for dll in dlls:
+        dll_base_name = dll.name.split("\\")[-1]
+        for function_name, function_address in dll.imports.items():
+            for name in generate_symbols(dll_base_name, function_name, include_dll=True):
+                yield Import(name), AbsoluteVirtualAddress(function_address)
+
+
+def extract_features(report: DrakvufReport) -> Iterator[tuple[Feature, Address]]:
+    for handler in FILE_HANDLERS:
+        for feature, addr in handler(report):
+            yield feature, addr
+
+
+FILE_HANDLERS = (
+    # TODO(yelhamer): extract more file features from other DRAKVUF plugins
+    # https://github.com/mandiant/capa/issues/2169
+    extract_import_names,
+)

capa/features/extractors/drakvuf/global_.py

@@ -0,0 +1,51 @@
+# Copyright 2024 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+import logging
+from typing import Iterator
+
+from capa.features.common import OS, FORMAT_PE, ARCH_AMD64, OS_WINDOWS, Arch, Format, Feature
+from capa.features.address import NO_ADDRESS, Address
+from capa.features.extractors.drakvuf.models import DrakvufReport
+
+logger = logging.getLogger(__name__)
+
+
+def extract_format(report: DrakvufReport) -> Iterator[tuple[Feature, Address]]:
+    # DRAKVUF sandbox currently supports only Windows as the guest: https://drakvuf-sandbox.readthedocs.io/en/latest/usage/getting_started.html
+    yield Format(FORMAT_PE), NO_ADDRESS
+
+
+def extract_os(report: DrakvufReport) -> Iterator[tuple[Feature, Address]]:
+    # DRAKVUF sandbox currently supports only PE files: https://drakvuf-sandbox.readthedocs.io/en/latest/usage/getting_started.html
+    yield OS(OS_WINDOWS), NO_ADDRESS
+
+
+def extract_arch(report: DrakvufReport) -> Iterator[tuple[Feature, Address]]:
+    # DRAKVUF sandbox currently supports only x64 Windows as the guest: https://drakvuf-sandbox.readthedocs.io/en/latest/usage/getting_started.html
+    yield Arch(ARCH_AMD64), NO_ADDRESS
+
+
+def extract_features(report: DrakvufReport) -> Iterator[tuple[Feature, Address]]:
+    for global_handler in GLOBAL_HANDLER:
+        for feature, addr in global_handler(report):
+            yield feature, addr
+
+
+GLOBAL_HANDLER = (
+    extract_format,
+    extract_os,
+    extract_arch,
+)

capa/features/extractors/drakvuf/helpers.py

@@ -0,0 +1,45 @@
+# Copyright 2024 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+import itertools
+
+from capa.features.address import ThreadAddress, ProcessAddress
+from capa.features.extractors.drakvuf.models import Call, DrakvufReport
+
+
+def index_calls(report: DrakvufReport) -> dict[ProcessAddress, dict[ThreadAddress, list[Call]]]:
+    # this method organizes calls into processes and threads, and then sorts them based on
+    # timestamp so that we can address individual calls per index (CallAddress requires call index)
+    result: dict[ProcessAddress, dict[ThreadAddress, list[Call]]] = {}
+    for call in itertools.chain(report.syscalls, report.apicalls):
+        if call.pid == 0:
+            # DRAKVUF captures api/native calls from all processes running on the system.
+            # we ignore the pid 0 since it's a system process and it's unlikely for it to
+            # be hijacked or so on, in addition to capa addresses not supporting null pids
+            continue
+        proc_addr = ProcessAddress(pid=call.pid, ppid=call.ppid)
+        thread_addr = ThreadAddress(process=proc_addr, tid=call.tid)
+        if proc_addr not in result:
+            result[proc_addr] = {}
+        if thread_addr not in result[proc_addr]:
+            result[proc_addr][thread_addr] = []
+
+        result[proc_addr][thread_addr].append(call)
+
+    for proc, threads in result.items():
+        for thread in threads:
+            result[proc][thread].sort(key=lambda call: call.timestamp)
+
+    return result

capa/features/extractors/drakvuf/models.py

@@ -0,0 +1,144 @@
+# Copyright 2024 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import logging
+from typing import Any, Iterator
+
+from pydantic import Field, BaseModel, ConfigDict, model_validator
+
+logger = logging.getLogger(__name__)
+
+
+REQUIRED_SYSCALL_FIELD_NAMES = {
+    "Plugin",
+    "TimeStamp",
+    "PID",
+    "PPID",
+    "TID",
+    "UserName",
+    "UserId",
+    "ProcessName",
+    "Method",
+    "EventUID",
+    "Module",
+    "vCPU",
+    "CR3",
+    "Syscall",
+    "NArgs",
+}
+
+
+class ConciseModel(BaseModel):
+    ConfigDict(extra="ignore")
+
+
+class DiscoveredDLL(ConciseModel):
+    plugin_name: str = Field(alias="Plugin")
+    event: str = Field(alias="Event")
+    name: str = Field(alias="DllName")
+    pid: int = Field(alias="PID")
+
+
+class LoadedDLL(ConciseModel):
+    plugin_name: str = Field(alias="Plugin")
+    event: str = Field(alias="Event")
+    name: str = Field(alias="DllName")
+    imports: dict[str, int] = Field(alias="Rva")
+
+
+class Call(ConciseModel):
+    plugin_name: str = Field(alias="Plugin")
+    timestamp: str = Field(alias="TimeStamp")
+    process_name: str = Field(alias="ProcessName")
+    ppid: int = Field(alias="PPID")
+    pid: int = Field(alias="PID")
+    tid: int = Field(alias="TID")
+    name: str = Field(alias="Method")
+    arguments: dict[str, str]
+
+
+class WinApiCall(Call):
+    # This class models Windows API calls captured by DRAKVUF (DLLs, etc.).
+    arguments: dict[str, str] = Field(alias="Arguments")
+    event: str = Field(alias="Event")
+    return_value: str = Field(alias="ReturnValue")
+
+    @model_validator(mode="before")
+    @classmethod
+    def build_arguments(cls, values: dict[str, Any]) -> dict[str, Any]:
+        args = values["Arguments"]
+        values["Arguments"] = dict(arg.split("=", 1) for arg in args)
+        return values
+
+
+class SystemCall(Call):
+    # This class models native Windows API calls captured by DRAKVUF.
+    # Schema: {
+    #   "Plugin": "syscall",
+    #   "TimeStamp": "1716999134.582553",
+    #   "PID": 3888, "PPID": 2852, "TID": 368, "UserName": "SessionID", "UserId": 2,
+    #   "ProcessName": "\\Device\\HarddiskVolume2\\Windows\\explorer.exe",
+    #   "Method": "NtSetIoCompletionEx",
+    #   "EventUID": "0x27",
+    #   "Module": "nt",
+    #   "vCPU": 0,
+    #   "CR3": "0x119b1002",
+    #   "Syscall": 419,
+    #   "NArgs": 6,
+    #   "IoCompletionHandle": "0xffffffff80001ac0", "IoCompletionReserveHandle": "0xffffffff8000188c",
+    #   "KeyContext": "0x0", "ApcContext": "0x2", "IoStatus": "0x7ffb00000000", "IoStatusInformation": "0x0"
+    # }
+    # The keys up until "NArgs" are common to all the native calls that DRAKVUF reports, with
+    # the remaining keys representing the call's specific arguments.
+    syscall_number: int = Field(alias="Syscall")
+    module: str = Field(alias="Module")
+    nargs: int = Field(alias="NArgs")
+
+    @model_validator(mode="before")
+    @classmethod
+    def build_extra(cls, values: dict[str, Any]) -> dict[str, Any]:
+        # DRAKVUF stores argument names and values as entries in the syscall's entry.
+        # This model validator collects those arguments into a list in the model.
+        values["arguments"] = {
+            name: value for name, value in values.items() if name not in REQUIRED_SYSCALL_FIELD_NAMES
+        }
+        return values
+
+
+class DrakvufReport(ConciseModel):
+    syscalls: list[SystemCall] = []
+    apicalls: list[WinApiCall] = []
+    discovered_dlls: list[DiscoveredDLL] = []
+    loaded_dlls: list[LoadedDLL] = []
+
+    @classmethod
+    def from_raw_report(cls, entries: Iterator[dict]) -> "DrakvufReport":
+        report = cls()
+
+        for entry in entries:
+            plugin = entry.get("Plugin")
+            # TODO(yelhamer): add support for more DRAKVUF plugins
+            # https://github.com/mandiant/capa/issues/2181
+            if plugin == "syscall":
+                report.syscalls.append(SystemCall(**entry))
+            elif plugin == "apimon":
+                event = entry.get("Event")
+                if event == "api_called":
+                    report.apicalls.append(WinApiCall(**entry))
+                elif event == "dll_loaded":
+                    report.loaded_dlls.append(LoadedDLL(**entry))
+                elif event == "dll_discovered":
+                    report.discovered_dlls.append(DiscoveredDLL(**entry))
+
+        return report

capa/features/extractors/drakvuf/process.py

@@ -0,0 +1,47 @@
+# Copyright 2024 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+import logging
+from typing import Iterator
+
+from capa.features.common import String, Feature
+from capa.features.address import Address, ThreadAddress, ProcessAddress
+from capa.features.extractors.base_extractor import ThreadHandle, ProcessHandle
+from capa.features.extractors.drakvuf.models import Call
+
+logger = logging.getLogger(__name__)
+
+
+def get_threads(
+    calls: dict[ProcessAddress, dict[ThreadAddress, list[Call]]], ph: ProcessHandle
+) -> Iterator[ThreadHandle]:
+    """
+    Get the threads associated with a given process.
+    """
+    for thread_addr in calls[ph.address]:
+        yield ThreadHandle(address=thread_addr, inner={})
+
+
+def extract_process_name(ph: ProcessHandle) -> Iterator[tuple[Feature, Address]]:
+    yield String(ph.inner["process_name"]), ph.address
+
+
+def extract_features(ph: ProcessHandle) -> Iterator[tuple[Feature, Address]]:
+    for handler in PROCESS_HANDLERS:
+        for feature, addr in handler(ph):
+            yield feature, addr
+
+
+PROCESS_HANDLERS = (extract_process_name,)

capa/features/extractors/drakvuf/thread.py

@@ -0,0 +1,31 @@
+# Copyright 2024 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+import logging
+from typing import Iterator
+
+from capa.features.address import ThreadAddress, ProcessAddress, DynamicCallAddress
+from capa.features.extractors.base_extractor import CallHandle, ThreadHandle, ProcessHandle
+from capa.features.extractors.drakvuf.models import Call
+
+logger = logging.getLogger(__name__)
+
+
+def get_calls(
+    sorted_calls: dict[ProcessAddress, dict[ThreadAddress, list[Call]]], ph: ProcessHandle, th: ThreadHandle
+) -> Iterator[CallHandle]:
+    for i, call in enumerate(sorted_calls[ph.address][th.address]):
+        call_addr = DynamicCallAddress(thread=th.address, id=i)
+        yield CallHandle(address=call_addr, inner=call)

capa/features/extractors/elf.py

@@ -1,16 +1,23 @@
-# Copyright (C) 2021 Mandiant, Inc. All Rights Reserved.
+# Copyright 2021 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# You may obtain a copy of the License at
-# Unless required by applicable law or agreed to in writing, software distributed under the License
+#
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#     http://www.apache.org/licenses/LICENSE-2.0
-# See the License for the specific language governing permissions and limitations under the License.
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 import struct
 import logging
 import itertools
 import collections
 from enum import Enum
-from typing import TYPE_CHECKING, Set, Dict, List, Tuple, BinaryIO, Iterator, Optional
+from typing import TYPE_CHECKING, BinaryIO, Iterator, Optional
 from dataclasses import dataclass
 
 if TYPE_CHECKING:
@@ -310,6 +317,9 @@ class ELF:
         98: "TPC",
         99: "SNP1K",
         100: "ST200",
+        # https://www.sco.com/developers/gabi/latest/ch4.eheader.html
+        183: "aarch64",
+        243: "riscv",
     }
 
     @property
@@ -391,7 +401,7 @@ class ELF:
             return read_cstr(phdr.buf, 0)
 
     @property
-    def versions_needed(self) -> Dict[str, Set[str]]:
+    def versions_needed(self) -> dict[str, set[str]]:
         # symbol version requirements are stored in the .gnu.version_r section,
         # which has type SHT_GNU_verneed (0x6ffffffe).
         #
@@ -449,7 +459,7 @@ class ELF:
         return {}
 
     @property
-    def dynamic_entries(self) -> Iterator[Tuple[int, int]]:
+    def dynamic_entries(self) -> Iterator[tuple[int, int]]:
         """
         read the entries from the dynamic section,
         yielding the tag and value for each entry.
@@ -544,7 +554,7 @@ class ELF:
                 logger.warning("failed to read DT_NEEDED entry: %s", str(e))
 
     @property
-    def symtab(self) -> Optional[Tuple[Shdr, Shdr]]:
+    def symtab(self) -> Optional[tuple[Shdr, Shdr]]:
         """
         fetch the Shdr for the symtab and the associated strtab.
         """
@@ -679,7 +689,7 @@ class SymTab:
         symtab: Shdr,
         strtab: Shdr,
     ) -> None:
-        self.symbols: List[Symbol] = []
+        self.symbols: list[Symbol] = []
 
         self.symtab = symtab
         self.strtab = strtab
@@ -1078,7 +1088,7 @@ def guess_os_from_go_buildinfo(elf: ELF) -> Optional[OS]:
     #  and the 32-byte header is followed by varint-prefixed string data
     #  for the two string values we care about.
     # https://github.com/mandiant/GoReSym/blob/0860a1b1b4f3495e9fb7e71eb4386bf3e0a7c500/buildinfo/buildinfo.go#L185-L193
-    BUILDINFO_MAGIC = b"\xFF Go buildinf:"
+    BUILDINFO_MAGIC = b"\xff Go buildinf:"
 
     try:
         index = buf.index(BUILDINFO_MAGIC)

capa/features/extractors/elffile.py

@@ -1,17 +1,23 @@
-# Copyright (C) 2021 Mandiant, Inc. All Rights Reserved.
+# Copyright 2021 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# You may obtain a copy of the License at
-# Unless required by applicable law or agreed to in writing, software distributed under the License
+#
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#     http://www.apache.org/licenses/LICENSE-2.0
-# See the License for the specific language governing permissions and limitations under the License.
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 import io
 import logging
-from typing import Tuple, Iterator
+from typing import Iterator
 from pathlib import Path
 
-from elftools.elf.elffile import ELFFile, SymbolTableSection
+from elftools.elf.elffile import ELFFile, DynamicSegment, SymbolTableSection
-from elftools.elf.relocation import RelocationSection
 
 import capa.features.extractors.common
 from capa.features.file import Export, Import, Section
@@ -47,17 +53,46 @@ def extract_file_export_names(elf: ELFFile, **kwargs):
 
             yield Export(symbol.name), AbsoluteVirtualAddress(symbol.entry.st_value)
 
-
+    for segment in elf.iter_segments():
-def extract_file_import_names(elf: ELFFile, **kwargs):
+        if not isinstance(segment, DynamicSegment):
-    # Create a dictionary to store symbol names by their index
-    symbol_names = {}
-
-    # Extract symbol names and store them in the dictionary
-    for section in elf.iter_sections():
-        if not isinstance(section, SymbolTableSection):
             continue
 
-        for _, symbol in enumerate(section.iter_symbols()):
+        tab_ptr, tab_offset = segment.get_table_offset("DT_SYMTAB")
+        if tab_ptr is None or tab_offset is None:
+            logger.debug("Dynamic segment doesn't contain DT_SYMTAB")
+            continue
+
+        logger.debug("Dynamic segment contains %s symbols: ", segment.num_symbols())
+
+        for symbol in segment.iter_symbols():
+            # The following conditions are based on the following article
+            # http://www.m4b.io/elf/export/binary/analysis/2015/05/25/what-is-an-elf-export.html
+            if not symbol.name:
+                continue
+            if symbol.entry.st_info.type not in ["STT_FUNC", "STT_OBJECT", "STT_IFUNC"]:
+                continue
+            if symbol.entry.st_value == 0:
+                continue
+            if symbol.entry.st_shndx == "SHN_UNDEF":
+                continue
+
+            yield Export(symbol.name), AbsoluteVirtualAddress(symbol.entry.st_value)
+
+
+def extract_file_import_names(elf: ELFFile, **kwargs):
+    symbol_name_by_index: dict[int, str] = {}
+
+    # Extract symbol names and store them in the dictionary
+    for segment in elf.iter_segments():
+        if not isinstance(segment, DynamicSegment):
+            continue
+
+        tab_ptr, tab_offset = segment.get_table_offset("DT_SYMTAB")
+        if tab_ptr is None or tab_offset is None:
+            logger.debug("Dynamic segment doesn't contain DT_SYMTAB")
+            continue
+
+        for i, symbol in enumerate(segment.iter_symbols()):
             # The following conditions are based on the following article
             # http://www.m4b.io/elf/export/binary/analysis/2015/05/25/what-is-an-elf-export.html
             if not symbol.name:
@@ -71,23 +106,37 @@ def extract_file_import_names(elf: ELFFile, **kwargs):
             if symbol.entry.st_name == 0:
                 continue
 
-            symbol_names[_] = symbol.name
+            symbol_name_by_index[i] = symbol.name
 
-    for section in elf.iter_sections():
+    for segment in elf.iter_segments():
-        if not isinstance(section, RelocationSection):
+        if not isinstance(segment, DynamicSegment):
             continue
 
-        if section["sh_entsize"] == 0:
+        relocation_tables = segment.get_relocation_tables()
-            logger.debug("Symbol table '%s' has a sh_entsize of zero!", section.name)
+        logger.debug("Dynamic Segment contains %s relocation tables:", len(relocation_tables))
+
+        for relocation_table in relocation_tables.values():
+            relocations = []
+            for i in range(relocation_table.num_relocations()):
+                try:
+                    relocations.append(relocation_table.get_relocation(i))
+                except TypeError:
+                    # ELF is corrupt and the relocation table is invalid,
+                    # so stop processing it.
+                    break
+
+            for relocation in relocations:
+                if "r_info_sym" not in relocation.entry or "r_offset" not in relocation.entry:
                     continue
 
-        logger.debug("Symbol table '%s' contains %s entries:", section.name, section.num_relocations())
+                symbol_address: int = relocation["r_offset"]
+                symbol_index: int = relocation["r_info_sym"]
 
-        for relocation in section.iter_relocations():
+                if symbol_index not in symbol_name_by_index:
-            # Extract the symbol name from the symbol table using the symbol index in the relocation
-            if relocation["r_info_sym"] not in symbol_names:
                     continue
-            yield Import(symbol_names[relocation["r_info_sym"]]), FileOffsetAddress(relocation["r_offset"])
+                symbol_name = symbol_name_by_index[symbol_index]
+
+                yield Import(symbol_name), FileOffsetAddress(symbol_address)
 
 
 def extract_file_section_names(elf: ELFFile, **kwargs):
@@ -122,11 +171,15 @@ def extract_file_arch(elf: ELFFile, **kwargs):
         yield Arch("i386"), NO_ADDRESS
     elif arch == "x64":
         yield Arch("amd64"), NO_ADDRESS
+    elif arch == "ARM":
+        yield Arch("arm"), NO_ADDRESS
+    elif arch == "AArch64":
+        yield Arch("aarch64"), NO_ADDRESS
     else:
         logger.warning("unsupported architecture: %s", arch)
 
 
-def extract_file_features(elf: ELFFile, buf: bytes) -> Iterator[Tuple[Feature, int]]:
+def extract_file_features(elf: ELFFile, buf: bytes) -> Iterator[tuple[Feature, int]]:
     for file_handler in FILE_HANDLERS:
         for feature, addr in file_handler(elf=elf, buf=buf):  # type: ignore
             yield feature, addr
@@ -142,7 +195,7 @@ FILE_HANDLERS = (
 )
 
 
-def extract_global_features(elf: ELFFile, buf: bytes) -> Iterator[Tuple[Feature, int]]:
+def extract_global_features(elf: ELFFile, buf: bytes) -> Iterator[tuple[Feature, int]]:
     for global_handler in GLOBAL_HANDLERS:
         for feature, addr in global_handler(elf=elf, buf=buf):  # type: ignore
             yield feature, addr

capa/features/extractors/ghidra/basicblock.py

@@ -1,14 +1,21 @@
-# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
+# Copyright 2023 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# You may obtain a copy of the License at
-# Unless required by applicable law or agreed to in writing, software distributed under the License
+#
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#     http://www.apache.org/licenses/LICENSE-2.0
-# See the License for the specific language governing permissions and limitations under the License.
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 
 import string
 import struct
-from typing import Tuple, Iterator
+from typing import Iterator
 
 import ghidra
 from ghidra.program.model.lang import OperandType
@@ -97,7 +104,7 @@ def _bb_has_tight_loop(bb: ghidra.program.model.block.CodeBlock):
     return False
 
 
-def extract_bb_stackstring(fh: FunctionHandle, bbh: BBHandle) -> Iterator[Tuple[Feature, Address]]:
+def extract_bb_stackstring(fh: FunctionHandle, bbh: BBHandle) -> Iterator[tuple[Feature, Address]]:
     """extract stackstring indicators from basic block"""
     bb: ghidra.program.model.block.CodeBlock = bbh.inner
 
@@ -105,7 +112,7 @@ def extract_bb_stackstring(fh: FunctionHandle, bbh: BBHandle) -> Iterator[Tuple[
         yield Characteristic("stack string"), bbh.address
 
 
-def extract_bb_tight_loop(fh: FunctionHandle, bbh: BBHandle) -> Iterator[Tuple[Feature, Address]]:
+def extract_bb_tight_loop(fh: FunctionHandle, bbh: BBHandle) -> Iterator[tuple[Feature, Address]]:
     """check basic block for tight loop indicators"""
     bb: ghidra.program.model.block.CodeBlock = bbh.inner
 
@@ -119,7 +126,7 @@ BASIC_BLOCK_HANDLERS = (
 )
 
 
-def extract_features(fh: FunctionHandle, bbh: BBHandle) -> Iterator[Tuple[Feature, Address]]:
+def extract_features(fh: FunctionHandle, bbh: BBHandle) -> Iterator[tuple[Feature, Address]]:
     """
     extract features from the given basic block.
 
@@ -127,7 +134,7 @@ def extract_features(fh: FunctionHandle, bbh: BBHandle) -> Iterator[Tuple[Featur
         bb: the basic block to process.
 
     yields:
-      Tuple[Feature, int]: the features and their location found in this basic block.
+      tuple[Feature, int]: the features and their location found in this basic block.
     """
     yield BasicBlock(), bbh.address
     for bb_handler in BASIC_BLOCK_HANDLERS:

capa/features/extractors/ghidra/extractor.py

@@ -1,11 +1,18 @@
-# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
+# Copyright 2023 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# You may obtain a copy of the License at
-# Unless required by applicable law or agreed to in writing, software distributed under the License
+#
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#     http://www.apache.org/licenses/LICENSE-2.0
-# See the License for the specific language governing permissions and limitations under the License.
+#
-from typing import List, Tuple, Iterator
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from typing import Iterator
 
 import capa.features.extractors.ghidra.file
 import capa.features.extractors.ghidra.insn
@@ -40,7 +47,7 @@ class GhidraFeatureExtractor(StaticFeatureExtractor):
             )
         )
 
-        self.global_features: List[Tuple[Feature, Address]] = []
+        self.global_features: list[tuple[Feature, Address]] = []
         self.global_features.extend(capa.features.extractors.ghidra.file.extract_file_format())
         self.global_features.extend(capa.features.extractors.ghidra.global_.extract_os())
         self.global_features.extend(capa.features.extractors.ghidra.global_.extract_arch())
@@ -73,7 +80,7 @@ class GhidraFeatureExtractor(StaticFeatureExtractor):
         func = getFunctionContaining(toAddr(addr))  # type: ignore [name-defined] # noqa: F821
         return FunctionHandle(address=AbsoluteVirtualAddress(func.getEntryPoint().getOffset()), inner=func)
 
-    def extract_function_features(self, fh: FunctionHandle) -> Iterator[Tuple[Feature, Address]]:
+    def extract_function_features(self, fh: FunctionHandle) -> Iterator[tuple[Feature, Address]]:
         yield from capa.features.extractors.ghidra.function.extract_features(fh)
 
     def get_basic_blocks(self, fh: FunctionHandle) -> Iterator[BBHandle]:
@@ -81,7 +88,7 @@ class GhidraFeatureExtractor(StaticFeatureExtractor):
 
         yield from ghidra_helpers.get_function_blocks(fh)
 
-    def extract_basic_block_features(self, fh: FunctionHandle, bbh: BBHandle) -> Iterator[Tuple[Feature, Address]]:
+    def extract_basic_block_features(self, fh: FunctionHandle, bbh: BBHandle) -> Iterator[tuple[Feature, Address]]:
         yield from capa.features.extractors.ghidra.basicblock.extract_features(fh, bbh)
 
     def get_instructions(self, fh: FunctionHandle, bbh: BBHandle) -> Iterator[InsnHandle]:

capa/features/extractors/ghidra/file.py

@@ -1,13 +1,20 @@
-# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
+# Copyright 2023 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# You may obtain a copy of the License at
-# Unless required by applicable law or agreed to in writing, software distributed under the License
+#
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#     http://www.apache.org/licenses/LICENSE-2.0
-# See the License for the specific language governing permissions and limitations under the License.
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 import re
 import struct
-from typing import List, Tuple, Iterator
+from typing import Iterator
 
 from ghidra.program.model.symbol import SourceType, SymbolType
 
@@ -22,7 +29,7 @@ from capa.features.address import NO_ADDRESS, Address, FileOffsetAddress, Absolu
 MAX_OFFSET_PE_AFTER_MZ = 0x200
 
 
-def find_embedded_pe(block_bytez: bytes, mz_xor: List[Tuple[bytes, bytes, int]]) -> Iterator[Tuple[int, int]]:
+def find_embedded_pe(block_bytez: bytes, mz_xor: list[tuple[bytes, bytes, int]]) -> Iterator[tuple[int, int]]:
     """check segment for embedded PE
 
     adapted for Ghidra from:
@@ -60,11 +67,11 @@ def find_embedded_pe(block_bytez: bytes, mz_xor: List[Tuple[bytes, bytes, int]])
             yield off, i
 
 
-def extract_file_embedded_pe() -> Iterator[Tuple[Feature, Address]]:
+def extract_file_embedded_pe() -> Iterator[tuple[Feature, Address]]:
     """extract embedded PE features"""
 
     # pre-compute XOR pairs
-    mz_xor: List[Tuple[bytes, bytes, int]] = [
+    mz_xor: list[tuple[bytes, bytes, int]] = [
         (
             capa.features.extractors.helpers.xor_static(b"MZ", i),
             capa.features.extractors.helpers.xor_static(b"PE", i),
@@ -84,14 +91,14 @@ def extract_file_embedded_pe() -> Iterator[Tuple[Feature, Address]]:
             yield Characteristic("embedded pe"), FileOffsetAddress(ea)
 
 
-def extract_file_export_names() -> Iterator[Tuple[Feature, Address]]:
+def extract_file_export_names() -> Iterator[tuple[Feature, Address]]:
     """extract function exports"""
     st = currentProgram().getSymbolTable()  # type: ignore [name-defined] # noqa: F821
     for addr in st.getExternalEntryPointIterator():
         yield Export(st.getPrimarySymbol(addr).getName()), AbsoluteVirtualAddress(addr.getOffset())
 
 
-def extract_file_import_names() -> Iterator[Tuple[Feature, Address]]:
+def extract_file_import_names() -> Iterator[tuple[Feature, Address]]:
     """extract function imports
 
     1. imports by ordinal:
@@ -116,14 +123,14 @@ def extract_file_import_names() -> Iterator[Tuple[Feature, Address]]:
             yield Import(name), AbsoluteVirtualAddress(addr)
 
 
-def extract_file_section_names() -> Iterator[Tuple[Feature, Address]]:
+def extract_file_section_names() -> Iterator[tuple[Feature, Address]]:
     """extract section names"""
 
     for block in currentProgram().getMemory().getBlocks():  # type: ignore [name-defined] # noqa: F821
         yield Section(block.getName()), AbsoluteVirtualAddress(block.getStart().getOffset())
 
 
-def extract_file_strings() -> Iterator[Tuple[Feature, Address]]:
+def extract_file_strings() -> Iterator[tuple[Feature, Address]]:
     """extract ASCII and UTF-16 LE strings"""
 
     for block in currentProgram().getMemory().getBlocks():  # type: ignore [name-defined] # noqa: F821
@@ -141,7 +148,7 @@ def extract_file_strings() -> Iterator[Tuple[Feature, Address]]:
             yield String(s.s), FileOffsetAddress(offset)
 
 
-def extract_file_function_names() -> Iterator[Tuple[Feature, Address]]:
+def extract_file_function_names() -> Iterator[tuple[Feature, Address]]:
     """
     extract the names of statically-linked library functions.
     """
@@ -162,7 +169,7 @@ def extract_file_function_names() -> Iterator[Tuple[Feature, Address]]:
                 yield FunctionName(name[1:]), addr
 
 
-def extract_file_format() -> Iterator[Tuple[Feature, Address]]:
+def extract_file_format() -> Iterator[tuple[Feature, Address]]:
     ef = currentProgram().getExecutableFormat()  # type: ignore [name-defined] # noqa: F821
     if "PE" in ef:
         yield Format(FORMAT_PE), NO_ADDRESS
@@ -175,7 +182,7 @@ def extract_file_format() -> Iterator[Tuple[Feature, Address]]:
         raise NotImplementedError(f"unexpected file format: {ef}")
 
 
-def extract_features() -> Iterator[Tuple[Feature, Address]]:
+def extract_features() -> Iterator[tuple[Feature, Address]]:
     """extract file features"""
     for file_handler in FILE_HANDLERS:
         for feature, addr in file_handler():

capa/features/extractors/ghidra/function.py

@@ -1,11 +1,18 @@
-# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
+# Copyright 2023 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# You may obtain a copy of the License at
-# Unless required by applicable law or agreed to in writing, software distributed under the License
+#
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#     http://www.apache.org/licenses/LICENSE-2.0
-# See the License for the specific language governing permissions and limitations under the License.
+#
-from typing import Tuple, Iterator
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from typing import Iterator
 
 import ghidra
 from ghidra.program.model.block import BasicBlockModel, SimpleBlockIterator
@@ -49,7 +56,7 @@ def extract_recursive_call(fh: FunctionHandle):
             yield Characteristic("recursive call"), AbsoluteVirtualAddress(f.getEntryPoint().getOffset())
 
 
-def extract_features(fh: FunctionHandle) -> Iterator[Tuple[Feature, Address]]:
+def extract_features(fh: FunctionHandle) -> Iterator[tuple[Feature, Address]]:
     for func_handler in FUNCTION_HANDLERS:
         for feature, addr in func_handler(fh):
             yield feature, addr

capa/features/extractors/ghidra/global_.py

@@ -1,13 +1,20 @@
-# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
+# Copyright 2023 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# You may obtain a copy of the License at
-# Unless required by applicable law or agreed to in writing, software distributed under the License
+#
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#     http://www.apache.org/licenses/LICENSE-2.0
-# See the License for the specific language governing permissions and limitations under the License.
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 import logging
 import contextlib
-from typing import Tuple, Iterator
+from typing import Iterator
 
 import capa.ghidra.helpers
 import capa.features.extractors.elf
@@ -18,7 +25,7 @@ from capa.features.address import NO_ADDRESS, Address
 logger = logging.getLogger(__name__)
 
 
-def extract_os() -> Iterator[Tuple[Feature, Address]]:
+def extract_os() -> Iterator[tuple[Feature, Address]]:
     format_name: str = currentProgram().getExecutableFormat()  # type: ignore [name-defined] # noqa: F821
 
     if "PE" in format_name:
@@ -45,7 +52,7 @@ def extract_os() -> Iterator[Tuple[Feature, Address]]:
         return
 
 
-def extract_arch() -> Iterator[Tuple[Feature, Address]]:
+def extract_arch() -> Iterator[tuple[Feature, Address]]:
     lang_id = currentProgram().getMetadata().get("Language ID")  # type: ignore [name-defined] # noqa: F821
 
     if "x86" in lang_id and "64" in lang_id:

capa/features/extractors/ghidra/helpers.py

@@ -1,11 +1,18 @@
-# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
+# Copyright 2023 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# You may obtain a copy of the License at
-# Unless required by applicable law or agreed to in writing, software distributed under the License
+#
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#     http://www.apache.org/licenses/LICENSE-2.0
-# See the License for the specific language governing permissions and limitations under the License.
+#
-from typing import Dict, List, Iterator
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from typing import Iterator
 
 import ghidra
 import java.lang
@@ -20,7 +27,7 @@ from capa.features.address import AbsoluteVirtualAddress
 from capa.features.extractors.base_extractor import BBHandle, InsnHandle, FunctionHandle
 
 
-def ints_to_bytes(bytez: List[int]) -> bytes:
+def ints_to_bytes(bytez: list[int]) -> bytes:
     """convert Java signed ints to Python bytes
 
     args:
@@ -83,10 +90,10 @@ def get_insn_in_range(bbh: BBHandle) -> Iterator[InsnHandle]:
         yield InsnHandle(address=AbsoluteVirtualAddress(insn.getAddress().getOffset()), inner=insn)
 
 
-def get_file_imports() -> Dict[int, List[str]]:
+def get_file_imports() -> dict[int, list[str]]:
     """get all import names & addrs"""
 
-    import_dict: Dict[int, List[str]] = {}
+    import_dict: dict[int, list[str]] = {}
 
     for f in currentProgram().getFunctionManager().getExternalFunctions():  # type: ignore [name-defined] # noqa: F821
         for r in f.getSymbol().getReferences():
@@ -110,7 +117,7 @@ def get_file_imports() -> Dict[int, List[str]]:
     return import_dict
 
 
-def get_file_externs() -> Dict[int, List[str]]:
+def get_file_externs() -> dict[int, list[str]]:
     """
     Gets function names & addresses of statically-linked library functions
 
@@ -124,7 +131,7 @@ def get_file_externs() -> Dict[int, List[str]]:
     - Note: See Symbol Table labels
     """
 
-    extern_dict: Dict[int, List[str]] = {}
+    extern_dict: dict[int, list[str]] = {}
 
     for sym in currentProgram().getSymbolTable().getAllSymbols(True):  # type: ignore [name-defined] # noqa: F821
         # .isExternal() misses more than this config for the function symbols
@@ -143,7 +150,7 @@ def get_file_externs() -> Dict[int, List[str]]:
     return extern_dict
 
 
-def map_fake_import_addrs() -> Dict[int, List[int]]:
+def map_fake_import_addrs() -> dict[int, list[int]]:
     """
     Map ghidra's fake import entrypoints to their
     real addresses
@@ -162,7 +169,7 @@ def map_fake_import_addrs() -> Dict[int, List[int]]:
     - 0x473090 -> PTR_CreateServiceW_00473090
     - 'EXTERNAL:00000025' -> External Address (ghidra.program.model.address.SpecialAddress)
     """
-    fake_dict: Dict[int, List[int]] = {}
+    fake_dict: dict[int, list[int]] = {}
 
     for f in currentProgram().getFunctionManager().getExternalFunctions():  # type: ignore [name-defined] # noqa: F821
         for r in f.getSymbol().getReferences():
@@ -174,9 +181,9 @@ def map_fake_import_addrs() -> Dict[int, List[int]]:
 
 def check_addr_for_api(
     addr: ghidra.program.model.address.Address,
-    fakes: Dict[int, List[int]],
+    fakes: dict[int, list[int]],
-    imports: Dict[int, List[str]],
+    imports: dict[int, list[str]],
-    externs: Dict[int, List[str]],
+    externs: dict[int, list[str]],
 ) -> bool:
     offset = addr.getOffset()
 

capa/features/extractors/ghidra/insn.py

@@ -1,11 +1,18 @@
-# Copyright (C) 2023 Mandiant, Inc. All Rights Reserved.
+# Copyright 2023 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# You may obtain a copy of the License at
-# Unless required by applicable law or agreed to in writing, software distributed under the License
+#
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#     http://www.apache.org/licenses/LICENSE-2.0
-# See the License for the specific language governing permissions and limitations under the License.
+#
-from typing import Any, Dict, Tuple, Iterator
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from typing import Any, Iterator
 
 import ghidra
 from ghidra.program.model.lang import OperandType
@@ -26,21 +33,21 @@ SECURITY_COOKIE_BYTES_DELTA = 0x40
 OPERAND_TYPE_DYNAMIC_ADDRESS = OperandType.DYNAMIC | OperandType.ADDRESS
 
 
-def get_imports(ctx: Dict[str, Any]) -> Dict[int, Any]:
+def get_imports(ctx: dict[str, Any]) -> dict[int, Any]:
     """Populate the import cache for this context"""
     if "imports_cache" not in ctx:
         ctx["imports_cache"] = capa.features.extractors.ghidra.helpers.get_file_imports()
     return ctx["imports_cache"]
 
 
-def get_externs(ctx: Dict[str, Any]) -> Dict[int, Any]:
+def get_externs(ctx: dict[str, Any]) -> dict[int, Any]:
     """Populate the externs cache for this context"""
     if "externs_cache" not in ctx:
         ctx["externs_cache"] = capa.features.extractors.ghidra.helpers.get_file_externs()
     return ctx["externs_cache"]
 
 
-def get_fakes(ctx: Dict[str, Any]) -> Dict[int, Any]:
+def get_fakes(ctx: dict[str, Any]) -> dict[int, Any]:
     """Populate the fake import addrs cache for this context"""
     if "fakes_cache" not in ctx:
         ctx["fakes_cache"] = capa.features.extractors.ghidra.helpers.map_fake_import_addrs()
@@ -48,7 +55,7 @@ def get_fakes(ctx: Dict[str, Any]) -> Dict[int, Any]:
 
 
 def check_for_api_call(
-    insn, externs: Dict[int, Any], fakes: Dict[int, Any], imports: Dict[int, Any], imp_or_ex: bool
+    insn, externs: dict[int, Any], fakes: dict[int, Any], imports: dict[int, Any], imp_or_ex: bool
 ) -> Iterator[Any]:
     """check instruction for API call
 
@@ -110,7 +117,7 @@ def check_for_api_call(
             yield info
 
 
-def extract_insn_api_features(fh: FunctionHandle, bb: BBHandle, ih: InsnHandle) -> Iterator[Tuple[Feature, Address]]:
+def extract_insn_api_features(fh: FunctionHandle, bb: BBHandle, ih: InsnHandle) -> Iterator[tuple[Feature, Address]]:
     insn: ghidra.program.database.code.InstructionDB = ih.inner
 
     if not capa.features.extractors.ghidra.helpers.is_call_or_jmp(insn):
@@ -131,7 +138,7 @@ def extract_insn_api_features(fh: FunctionHandle, bb: BBHandle, ih: InsnHandle)
             yield API(ext), ih.address
 
 
-def extract_insn_number_features(fh: FunctionHandle, bb: BBHandle, ih: InsnHandle) -> Iterator[Tuple[Feature, Address]]:
+def extract_insn_number_features(fh: FunctionHandle, bb: BBHandle, ih: InsnHandle) -> Iterator[tuple[Feature, Address]]:
     """
     parse instruction number features
     example:
@@ -186,7 +193,7 @@ def extract_insn_number_features(fh: FunctionHandle, bb: BBHandle, ih: InsnHandl
                 yield OperandOffset(i, const), addr
 
 
-def extract_insn_offset_features(fh: FunctionHandle, bb: BBHandle, ih: InsnHandle) -> Iterator[Tuple[Feature, Address]]:
+def extract_insn_offset_features(fh: FunctionHandle, bb: BBHandle, ih: InsnHandle) -> Iterator[tuple[Feature, Address]]:
     """
     parse instruction structure offset features
 
@@ -219,7 +226,7 @@ def extract_insn_offset_features(fh: FunctionHandle, bb: BBHandle, ih: InsnHandl
             yield OperandOffset(i, op_off), ih.address
 
 
-def extract_insn_bytes_features(fh: FunctionHandle, bb: BBHandle, ih: InsnHandle) -> Iterator[Tuple[Feature, Address]]:
+def extract_insn_bytes_features(fh: FunctionHandle, bb: BBHandle, ih: InsnHandle) -> Iterator[tuple[Feature, Address]]:
     """
     parse referenced byte sequences
 
@@ -234,7 +241,7 @@ def extract_insn_bytes_features(fh: FunctionHandle, bb: BBHandle, ih: InsnHandle
                 yield Bytes(extracted_bytes), ih.address
 
 
-def extract_insn_string_features(fh: FunctionHandle, bb: BBHandle, ih: InsnHandle) -> Iterator[Tuple[Feature, Address]]:
+def extract_insn_string_features(fh: FunctionHandle, bb: BBHandle, ih: InsnHandle) -> Iterator[tuple[Feature, Address]]:
     """
     parse instruction string features
 
@@ -249,7 +256,7 @@ def extract_insn_string_features(fh: FunctionHandle, bb: BBHandle, ih: InsnHandl
 
 def extract_insn_mnemonic_features(
     fh: FunctionHandle, bb: BBHandle, ih: InsnHandle
-) -> Iterator[Tuple[Feature, Address]]:
+) -> Iterator[tuple[Feature, Address]]:
     """parse instruction mnemonic features"""
     insn: ghidra.program.database.code.InstructionDB = ih.inner
 
@@ -258,7 +265,7 @@ def extract_insn_mnemonic_features(
 
 def extract_insn_obfs_call_plus_5_characteristic_features(
     fh: FunctionHandle, bb: BBHandle, ih: InsnHandle
-) -> Iterator[Tuple[Feature, Address]]:
+) -> Iterator[tuple[Feature, Address]]:
     """
     parse call $+5 instruction from the given instruction.
     """
@@ -279,7 +286,7 @@ def extract_insn_obfs_call_plus_5_characteristic_features(
 
 def extract_insn_segment_access_features(
     fh: FunctionHandle, bb: BBHandle, ih: InsnHandle
-) -> Iterator[Tuple[Feature, Address]]:
+) -> Iterator[tuple[Feature, Address]]:
     """parse instruction fs or gs access"""
     insn: ghidra.program.database.code.InstructionDB = ih.inner
 
@@ -294,7 +301,7 @@ def extract_insn_segment_access_features(
 
 def extract_insn_peb_access_characteristic_features(
     fh: FunctionHandle, bb: BBHandle, ih: InsnHandle
-) -> Iterator[Tuple[Feature, Address]]:
+) -> Iterator[tuple[Feature, Address]]:
     """parse instruction peb access
 
     fs:[0x30] on x86, gs:[0x60] on x64
@@ -310,7 +317,7 @@ def extract_insn_peb_access_characteristic_features(
 
 def extract_insn_cross_section_cflow(
     fh: FunctionHandle, bb: BBHandle, ih: InsnHandle
-) -> Iterator[Tuple[Feature, Address]]:
+) -> Iterator[tuple[Feature, Address]]:
     """inspect the instruction for a CALL or JMP that crosses section boundaries"""
     insn: ghidra.program.database.code.InstructionDB = ih.inner
 
@@ -364,7 +371,7 @@ def extract_function_calls_from(
     fh: FunctionHandle,
     bb: BBHandle,
     ih: InsnHandle,
-) -> Iterator[Tuple[Feature, Address]]:
+) -> Iterator[tuple[Feature, Address]]:
     """extract functions calls from features
 
     most relevant at the function scope, however, its most efficient to extract at the instruction scope
@@ -393,7 +400,7 @@ def extract_function_indirect_call_characteristic_features(
     fh: FunctionHandle,
     bb: BBHandle,
     ih: InsnHandle,
-) -> Iterator[Tuple[Feature, Address]]:
+) -> Iterator[tuple[Feature, Address]]:
     """extract indirect function calls (e.g., call eax or call dword ptr [edx+4])
     does not include calls like => call ds:dword_ABD4974
 
@@ -412,29 +419,28 @@ def extract_function_indirect_call_characteristic_features(
 def check_nzxor_security_cookie_delta(
     fh: ghidra.program.database.function.FunctionDB, insn: ghidra.program.database.code.InstructionDB
 ):
-    """Get the function containing the insn
+    """
-    Get the last block of the function that contains the insn
+    Get the first and last blocks of the function
-
+    Check if insn within first addr of first bb + delta
-    Check the bb containing the insn
+    Check if insn within last addr of last bb - delta
-    Check the last bb of the function containing the insn
     """
 
     model = SimpleBlockModel(currentProgram())  # type: ignore [name-defined] # noqa: F821
     insn_addr = insn.getAddress()
     func_asv = fh.getBody()
-    first_addr = func_asv.getMinAddress()
-    last_addr = func_asv.getMaxAddress()
 
-    if model.getFirstCodeBlockContaining(
+    first_addr = func_asv.getMinAddress()
-        first_addr, monitor()  # type: ignore [name-defined] # noqa: F821
-    ) == model.getFirstCodeBlockContaining(
-        last_addr, monitor()  # type: ignore [name-defined] # noqa: F821
-    ):
     if insn_addr < first_addr.add(SECURITY_COOKIE_BYTES_DELTA):
+        first_bb = model.getFirstCodeBlockContaining(first_addr, monitor())  # type: ignore [name-defined] # noqa: F821
+        if first_bb.contains(insn_addr):
             return True
-        else:
+
-            return insn_addr > last_addr.add(SECURITY_COOKIE_BYTES_DELTA * -1)
+    last_addr = func_asv.getMaxAddress()
-    else:
+    if insn_addr > last_addr.add(SECURITY_COOKIE_BYTES_DELTA * -1):
+        last_bb = model.getFirstCodeBlockContaining(last_addr, monitor())  # type: ignore [name-defined] # noqa: F821
+        if last_bb.contains(insn_addr):
+            return True
+
     return False
 
 
@@ -442,7 +448,7 @@ def extract_insn_nzxor_characteristic_features(
     fh: FunctionHandle,
     bb: BBHandle,
     ih: InsnHandle,
-) -> Iterator[Tuple[Feature, Address]]:
+) -> Iterator[tuple[Feature, Address]]:
     f: ghidra.program.database.function.FunctionDB = fh.inner
     insn: ghidra.program.database.code.InstructionDB = ih.inner
 
@@ -461,7 +467,7 @@ def extract_features(
     fh: FunctionHandle,
     bb: BBHandle,
     insn: InsnHandle,
-) -> Iterator[Tuple[Feature, Address]]:
+) -> Iterator[tuple[Feature, Address]]:
     for insn_handler in INSTRUCTION_HANDLERS:
         for feature, addr in insn_handler(fh, bb, insn):
             yield feature, addr

capa/features/extractors/helpers.py

@@ -1,14 +1,21 @@
-# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
+# Copyright 2020 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# You may obtain a copy of the License at
-# Unless required by applicable law or agreed to in writing, software distributed under the License
+#
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#     http://www.apache.org/licenses/LICENSE-2.0
-# See the License for the specific language governing permissions and limitations under the License.
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 
 import struct
 import builtins
-from typing import Tuple, Iterator
+from typing import Iterator
 
 MIN_STACKSTRING_LEN = 8
 
@@ -63,6 +70,7 @@ def generate_symbols(dll: str, symbol: str, include_dll=False) -> Iterator[str]:
     # trim extensions observed in dynamic traces
     dll = dll[0:-4] if dll.endswith(".dll") else dll
     dll = dll[0:-4] if dll.endswith(".drv") else dll
+    dll = dll[0:-3] if dll.endswith(".so") else dll
 
     if include_dll or is_ordinal(symbol):
         # ws2_32.#1
@@ -118,7 +126,7 @@ def twos_complement(val: int, bits: int) -> int:
         return val
 
 
-def carve_pe(pbytes: bytes, offset: int = 0) -> Iterator[Tuple[int, int]]:
+def carve_pe(pbytes: bytes, offset: int = 0) -> Iterator[tuple[int, int]]:
     """
     Generate (offset, key) tuples of embedded PEs
 

capa/features/extractors/ida/basicblock.py

@@ -1,14 +1,21 @@
-# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
+# Copyright 2020 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# You may obtain a copy of the License at
-# Unless required by applicable law or agreed to in writing, software distributed under the License
+#
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#     http://www.apache.org/licenses/LICENSE-2.0
-# See the License for the specific language governing permissions and limitations under the License.
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 
 import string
 import struct
-from typing import Tuple, Iterator
+from typing import Iterator
 
 import idaapi
 
@@ -80,19 +87,19 @@ def bb_contains_stackstring(f: idaapi.func_t, bb: idaapi.BasicBlock) -> bool:
     return False
 
 
-def extract_bb_stackstring(fh: FunctionHandle, bbh: BBHandle) -> Iterator[Tuple[Feature, Address]]:
+def extract_bb_stackstring(fh: FunctionHandle, bbh: BBHandle) -> Iterator[tuple[Feature, Address]]:
     """extract stackstring indicators from basic block"""
     if bb_contains_stackstring(fh.inner, bbh.inner):
         yield Characteristic("stack string"), bbh.address
 
 
-def extract_bb_tight_loop(fh: FunctionHandle, bbh: BBHandle) -> Iterator[Tuple[Feature, Address]]:
+def extract_bb_tight_loop(fh: FunctionHandle, bbh: BBHandle) -> Iterator[tuple[Feature, Address]]:
     """extract tight loop indicators from a basic block"""
     if capa.features.extractors.ida.helpers.is_basic_block_tight_loop(bbh.inner):
         yield Characteristic("tight loop"), bbh.address
 
 
-def extract_features(fh: FunctionHandle, bbh: BBHandle) -> Iterator[Tuple[Feature, Address]]:
+def extract_features(fh: FunctionHandle, bbh: BBHandle) -> Iterator[tuple[Feature, Address]]:
     """extract basic block features"""
     for bb_handler in BASIC_BLOCK_HANDLERS:
         for feature, addr in bb_handler(fh, bbh):

capa/features/extractors/ida/extractor.py

@@ -1,14 +1,20 @@
-# Copyright (C) 2021 Mandiant, Inc. All Rights Reserved.
+# Copyright 2021 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# You may obtain a copy of the License at
-# Unless required by applicable law or agreed to in writing, software distributed under the License
+#
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#     http://www.apache.org/licenses/LICENSE-2.0
-# See the License for the specific language governing permissions and limitations under the License.
+#
-from typing import List, Tuple, Iterator
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from typing import Iterator
 
 import idaapi
-import ida_nalt
 
 import capa.ida.helpers
 import capa.features.extractors.elf
@@ -32,10 +38,12 @@ class IdaFeatureExtractor(StaticFeatureExtractor):
     def __init__(self):
         super().__init__(
             hashes=SampleHashes(
-                md5=ida_nalt.retrieve_input_file_md5(), sha1="(unknown)", sha256=ida_nalt.retrieve_input_file_sha256()
+                md5=capa.ida.helpers.retrieve_input_file_md5(),
+                sha1="(unknown)",
+                sha256=capa.ida.helpers.retrieve_input_file_sha256(),
             )
         )
-        self.global_features: List[Tuple[Feature, Address]] = []
+        self.global_features: list[tuple[Feature, Address]] = []
         self.global_features.extend(capa.features.extractors.ida.file.extract_file_format())
         self.global_features.extend(capa.features.extractors.ida.global_.extract_os())
         self.global_features.extend(capa.features.extractors.ida.global_.extract_arch())
@@ -60,7 +68,7 @@ class IdaFeatureExtractor(StaticFeatureExtractor):
         f = idaapi.get_func(ea)
         return FunctionHandle(address=AbsoluteVirtualAddress(f.start_ea), inner=f)
 
-    def extract_function_features(self, fh: FunctionHandle) -> Iterator[Tuple[Feature, Address]]:
+    def extract_function_features(self, fh: FunctionHandle) -> Iterator[tuple[Feature, Address]]:
         yield from capa.features.extractors.ida.function.extract_features(fh)
 
     def get_basic_blocks(self, fh: FunctionHandle) -> Iterator[BBHandle]:
@@ -69,7 +77,7 @@ class IdaFeatureExtractor(StaticFeatureExtractor):
         for bb in ida_helpers.get_function_blocks(fh.inner):
             yield BBHandle(address=AbsoluteVirtualAddress(bb.start_ea), inner=bb)
 
-    def extract_basic_block_features(self, fh: FunctionHandle, bbh: BBHandle) -> Iterator[Tuple[Feature, Address]]:
+    def extract_basic_block_features(self, fh: FunctionHandle, bbh: BBHandle) -> Iterator[tuple[Feature, Address]]:
         yield from capa.features.extractors.ida.basicblock.extract_features(fh, bbh)
 
     def get_instructions(self, fh: FunctionHandle, bbh: BBHandle) -> Iterator[InsnHandle]:

capa/features/extractors/ida/file.py

@@ -1,19 +1,27 @@
-# Copyright (C) 2020 Mandiant, Inc. All Rights Reserved.
+# Copyright 2020 Google LLC
+#
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at: [package root]/LICENSE.txt
+# You may obtain a copy of the License at
-# Unless required by applicable law or agreed to in writing, software distributed under the License
+#
-#  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#     http://www.apache.org/licenses/LICENSE-2.0
-# See the License for the specific language governing permissions and limitations under the License.
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 
 import struct
-from typing import Tuple, Iterator
+from typing import Iterator
 
 import idc
 import idaapi
 import idautils
 import ida_entry
 
+import capa.ida.helpers
 import capa.features.extractors.common
 import capa.features.extractors.helpers
 import capa.features.extractors.strings
@@ -25,7 +33,7 @@ from capa.features.address import NO_ADDRESS, Address, FileOffsetAddress, Absolu
 MAX_OFFSET_PE_AFTER_MZ = 0x200
 
 
-def check_segment_for_pe(seg: idaapi.segment_t) -> Iterator[Tuple[int, int]]:
+def check_segment_for_pe(seg: idaapi.segment_t) -> Iterator[tuple[int, int]]:
     """check segment for embedded PE
 
     adapted for IDA from:
@@ -70,7 +78,7 @@ def check_segment_for_pe(seg: idaapi.segment_t) -> Iterator[Tuple[int, int]]:
             yield off, i
 
 
-def extract_file_embedded_pe() -> Iterator[Tuple[Feature, Address]]:
+def extract_file_embedded_pe() -> Iterator[tuple[Feature, Address]]:
     """extract embedded PE features
 
     IDA must load resource sections for this to be complete
@@ -82,7 +90,7 @@ def extract_file_embedded_pe() -> Iterator[Tuple[Feature, Address]]:
             yield Characteristic("embedded pe"), FileOffsetAddress(ea)
 
 
-def extract_file_export_names() -> Iterator[Tuple[Feature, Address]]:
+def extract_file_export_names() -> Iterator[tuple[Feature, Address]]:
     """extract function exports"""
     for _, ordinal, ea, name in idautils.Entries():
         forwarded_name = ida_entry.get_entry_forwarder(ordinal)
@@ -94,7 +102,7 @@ def extract_file_export_names() -> Iterator[Tuple[Feature, Address]]:
             yield Characteristic("forwarded export"), AbsoluteVirtualAddress(ea)
 
 
-def extract_file_import_names() -> Iterator[Tuple[Feature, Address]]:
+def extract_file_import_names() -> Iterator[tuple[Feature, Address]]:
     """extract function imports
 
     1. imports by ordinal:
@@ -130,7 +138,7 @@ def extract_file_import_names() -> Iterator[Tuple[Feature, Address]]:
         yield Import(info[1]), AbsoluteVirtualAddress(ea)
 
 
-def extract_file_section_names() -> Iterator[Tuple[Feature, Address]]:
+def extract_file_section_names() -> Iterator[tuple[Feature, Address]]:
     """extract section names
 
     IDA must load resource sections for this to be complete
@@ -141,7 +149,7 @@ def extract_file_section_names() -> Iterator[Tuple[Feature, Address]]:
         yield Section(idaapi.get_segm_name(seg)), AbsoluteVirtualAddress(seg.start_ea)
 
 
-def extract_file_strings() -> Iterator[Tuple[Feature, Address]]:
+def extract_file_strings() -> Iterator[tuple[Feature, Address]]:
     """extract ASCII and UTF-16 LE strings
 
     IDA must load resource sections for this to be complete
@@ -159,7 +167,7 @@ def extract_file_strings() -> Iterator[Tuple[Feature, Address]]:
             yield String(s.s), FileOffsetAddress(seg.start_ea + s.offset)
 
 
-def extract_file_function_names() -> Iterator[Tuple[Feature, Address]]:
+def extract_file_function_names() -> Iterator[tuple[Feature, Address]]:
     """
     extract the names of statically-linked library functions.
     """
@@ -176,21 +184,21 @@ def extract_file_function_names() -> Iterator[Tuple[Feature, Address]]:
                 yield FunctionName(name[1:]), addr
 
 
-def extract_file_format() -> Iterator[Tuple[Feature, Address]]:
+def extract_file_format() -> Iterator[tuple[Feature, Address]]:
-    file_info = idaapi.get_inf_structure()
+    filetype = capa.ida.helpers.get_filetype()
 
-    if file_info.filetype in (idaapi.f_PE, idaapi.f_COFF):
+    if filetype in (idaapi.f_PE, idaapi.f_COFF):
         yield Format(FORMAT_PE), NO_ADDRESS
-    elif file_info.filetype == idaapi.f_ELF:
+    elif filetype == idaapi.f_ELF:
         yield Format(FORMAT_ELF), NO_ADDRESS
-    elif file_info.filetype == idaapi.f_BIN:
+    elif filetype == idaapi.f_BIN:
         # no file type to return when processing a binary file, but we want to continue processing
         return
     else:
-        raise NotImplementedError(f"unexpected file format: {file_info.filetype}")
+        raise NotImplementedError(f"unexpected file format: {filetype}")
 
 
-def extract_features() -> Iterator[Tuple[Feature, Address]]:
+def extract_features() -> Iterator[tuple[Feature, Address]]:
     """extract file features"""
     for file_handler in FILE_HANDLERS:
         for feature, addr in file_handler():