python-tokenizers

maintainer xiota · 16 votes · scanned 2026-08-03 00:08:14.047287

LOW

Why flagged The package downloads source code from the project's official GitHub and two data files from reputable academic and project-specific domains; the non-standard hosts are for non-executable data, and the build uses standard Rust/Python tooling on the project's own source, posing minimal risk.

Triggered rules

LOW AI review downgraded a static finding llm_review

The static rules flagged this MEDIUM, but an AI model (qwen/qwen3-235b-a22b-07-25) reviewed the full PKGBUILD and judged it LOW (confidence 95%): The package downloads source code from the project's official GitHub and two data files from reputable academic and project-specific domains; the non-standard hosts are for non-executable data, and the build uses standard Rust/Python tooling on the project's own source, posing minimal risk.

1 higher static finding superseded - not the current verdict (shown for transparency)

MEDIUM source=() URL on a non-standard host source_untrusted_domain

One or more source=() URLs point to a host outside the trusted allowlist (github.com, gitlab.com, codeberg.org, pypi.org, …).

PKGBUILD:45 "norvig-big.txt"::"https://norvig.com/big.txt"
PKGBUILD:46 "roberta.json"::"https://huggingface.co/roberta-large/raw/main/tokenizer.json"

PKGBUILD

2 offending line(s) highlighted

      1# Maintainer: Daniel Bershatsky <bepshatsky@yandex.ru>
    
      2# Contributor: Filip Graliński <filipg@amu.edu.pl>
    
      3
    
      4: ${CARGO_HOME:=$SRCDEST/cargo-home}
    
      5: ${CARGO_TARGET_DIR:=target}
    
      6: ${RUSTUP_TOOLCHAIN:=stable}
    
      7
    
      8_module="tokenizers"
    
      9_pkgname="python-$_module"
    
      10pkgname="$_pkgname"
    
      11pkgver=0.23.1
    
      12pkgrel=1
    
      13pkgdesc='Fast State-of-the-Art Tokenizers optimized for Research and Production'
    
      14url="https://github.com/huggingface/tokenizers"
    
      15license=('Apache-2.0')
    
      16arch=('i686' 'x86_64')
    
      17
    
      18depends=('python')
    
      19makedepends=(
    
      20  'clang'
    
      21  'rust-bindgen'
    
      22  'python-build'
    
      23  'python-installer'
    
      24  'python-maturin'
    
      25  'python-setuptools-rust'
    
      26  'python-wheel'
    
      27)
    
      28checkdepends=(
    
      29  'python-datasets'
    
      30  #└─ 'python-huggingface-hub' # AUR
    
      31  #└─ 'python-multiprocess' # AUR
    
      32  'python-numpy'
    
      33  'python-pyarrow'
    
      34  'python-pytest'
    
      35  'python-requests'
    
      36  'python-setuptools-rust'
    
      37)
    
      38
    
      39options=('!lto')
    
      40
    
      41_pkgsrc="$_module-$pkgver"
    
      42_pkgext="tar.gz"
    
      43source=(
    
      44  "$_pkgsrc.$_pkgext"::"$url/archive/refs/tags/v$pkgver.$_pkgext"
    
      45  "norvig-big.txt"::"https://norvig.com/big.txt"
    
      46  "roberta.json"::"https://huggingface.co/roberta-large/raw/main/tokenizer.json"
    
      47)
    
      48sha256sums=('aa906ad27ece40261e075e171e4a8873c2c5cfdbb64205170735d425f214c7ef'
    
      49            'fa066c7d40f0f201ac4144e652aa62430e58a6b3805ec70650f678da5804e87b'
    
      50            '847bbeab6174d66a88898f729d52fa8d355fafe1bea101cf960dd404581df70e')
    
      51
    
      52_rust_env() {
    
      53  export CARGO_HOME CARGO_TARGET_DIR RUSTUP_TOOLCHAIN
    
      54  export GIT_DIR='.'
    
      55}
    
      56
    
      57prepare() (
    
      58  _rust_env
    
      59  cd "$_pkgsrc/bindings/python"
    
      60
    
      61  # fix typo
    
      62  sed -E -e 's@defaut@default@' -i Cargo.toml
    
      63
    
      64  cargo update
    
      65  cargo fetch --locked --target "$(rustc -vV | sed -n 's/host: //p')"
    
      66)
    
      67
    
      68build() (
    
      69  _rust_env
    
      70  cd "$_pkgsrc/bindings/python"
    
      71  cargo build --frozen --release
    
      72  python -m build --no-isolation --wheel
    
      73)
    
      74
    
      75check() {
    
      76  cd "$_pkgsrc/bindings/python"
    
      77
    
      78  # data
    
      79  install -Dm644 "$srcdir/roberta.json" "data/roberta.json"
    
      80  install -Dm644 "$srcdir/norvig-big.txt" "data/big.txt"
    
      81  head -100 data/big.txt > data/small.txt
    
      82
    
      83  # test in venv
    
      84  python -m venv --system-site-packages test-env
    
      85  test-env/bin/python -m installer dist/*.whl
    
      86  test-env/bin/python -m pytest -s -v tests || :
    
      87
    
      88  cargo test --no-default-features || :
    
      89}
    
      90
    
      91package() {
    
      92  cd "$_pkgsrc"
    
      93  install -Dm 644 LICENSE "$pkgdir/usr/share/licenses/$pkgname/LICENSE"
    
      94  cd "bindings/python"
    
      95  python -m installer --destdir="$pkgdir" dist/*.whl
    
      96}
    
      97

Scan history

Scanned at (UTC)	Severity	Rules
2026-08-03 00:08:14	LOW	2
2026-08-02 00:16:08	LOW	2
2026-08-01 00:11:18	LOW	2
2026-07-31 00:14:10	LOW	2
2026-07-30 00:17:23	LOW	2
2026-07-29 00:25:53	LOW	2
2026-07-28 00:07:28	LOW	2
2026-07-27 00:24:32	LOW	2
2026-07-26 00:07:32	LOW	2
2026-07-25 00:13:44	LOW	2
2026-07-24 00:02:28	LOW	2
2026-07-23 00:14:47	LOW	2
2026-07-22 00:29:32	LOW	2
2026-07-21 00:24:15	LOW	2
2026-07-20 00:19:49	LOW	2
2026-07-19 00:17:08	LOW	2
2026-07-18 00:14:48	LOW	2
2026-07-17 00:06:16	LOW	2
2026-07-16 00:05:41	LOW	2
2026-07-15 00:09:25	LOW	2