# Indian Digital Payments Stat Hub

Open stat hub for Indian digital payments, built only from official sources:
NPCI product statistics API + RBI PSI / bank-wise volumes / ATM-POS-Card statistics.

- Live hub: https://cashlessconsumer.zo.space/payments-hub (tabs: Explorer, Catalog, About)
- Query box: natural-language-ish catalog search — "business declines on UPI across months" → UPI uptime & incidents series
- API: https://cashlessconsumer.zo.space/api/payments-hub (datasets: summary, monthly, daily, quality, psi-indicators, psi-series, bankwise, atmpos, catalog with `q=` search)
- Data archive (CSVs + JSON slices + fetch/normalize scripts + catalog): https://zo.pub/cashlessconsumer/payments-stat-hub

## Layout

- `raw/npci/<product>/<stream>/<period>.json` — raw API responses (one file per fetch unit)
- `raw/npci/eco/<tab>/<side>/<YYYY-MM>.json` — UPI ecosystem deep-dive (BD/TD banks/PSPs, MCC, state-wise, apps, chargebacks, top-50)
- `raw/rbi/<source>/<Year-Month>.xlsx` — raw RBI Excel files
- `data/payments.duckdb` — normalized warehouse (npci_* + rbi_* tables, see below)
- `data/catalog.json` — machine-readable catalog: source URL, fetch script, coverage per dataset
- `data/quality.json` — monthly uptime %, incidents, bank downtime per rail
- `data/monthly.json`, `data/psi_indicators.json`, `data/bankwise_totals.json` — explorer slices
- `notes/2026-09-18-build-notes.md` — build log + source quirks

## DuckDB tables

- `npci_product_monthly` — product × month volume/value/banks (UPI, IMPS, BHIM, NETC)
- `npci_trended_monthly` — trended stats (UPI, NFS; from NPCI's trended tabs)
- `npci_daily` — daily volume/value (UPI, NFS)
- `npci_other` — uptime/downtime disclosures, member lists, vouchers
- `rbi_psi` — Payment System Indicators, long format (month, part, section, indicator, metric, unit, series_type, series_label, value), 2016-01 → (long format: indicator × month × value)
- `rbi_bankwise` — bank × rail × month inward/outward txn/value (ECS/NEFT/RTGS/Mobile)
- `rbi_atmposcard` — bank × month ATM/PoS/cards/micro-ATM infra
- `eco_upi_apps` — UPI app × month volumes (PhonePe/GPay/Paytm/BHIM/…)
- `eco_top50_member` — top-50 remitter/beneficiary banks: volume, approved %, business-decline %, technical-decline %
- `eco_top50_psp` — same, payer/payee PSP side
- `eco_top50_volval` — top-50 banks by volume & value
- `eco_mcc` — merchant-category volumes/values with share %
- `eco_statewise` — state/UT volumes/values with share %
- `eco_p2p_p2m` — P2P vs P2M split per month
- `eco_chargeback` — chargebacks received/represented/accepted per month
- `fetch_log` — every fetch unit with row count, status, source URL

## Refresh

```bash
python3 Skills/payments-fetch/scripts/npci_fetch.py           # all NPCI products
python3 Skills/payments-fetch/scripts/npci_fetch.py --product upi
python3 Skills/payments-fetch/scripts/rbi_fetch.py            # all RBI sources, current+prev year
python3 Skills/payments-fetch/scripts/rbi_fetch.py --only psi --years 2024 2025
python3 Skills/payments-fetch/scripts/npci_eco_fetch.py            # UPI ecosystem deep-dive (BD/TD, MCC, state, apps, chargebacks)
python3 Skills/payments-fetch/scripts/npci_eco_fetch.py --months 2026-08 2026-07
python3 Skills/payments-fetch/scripts/npci_normalize.py       # raw → payments.duckdb + exports
python3 Skills/payments-fetch/scripts/npci_eco_normalize.py   # eco raw → payments.duckdb
python3 Skills/payments-fetch/scripts/rbi_normalize.py        # xlsx → payments.duckdb
python3 Skills/payments-fetch/scripts/build_hub_data.py       # regenerate site exports + catalog (24 datasets)
```

All fetchers are idempotent (existing non-empty files skipped) and log to `raw/npci/manifest.jsonl` / `raw/rbi/fetch_log.jsonl`.

## Source quirks (verified 2026-09-18)

- NPCI is behind a bot wall (403 to curl) — fetcher drives headless Chromium and calls
  `/api/product-statistic/tabs` + `/tab/detail` inside the page session.
- NPCI's uptime/downtime API streams only serve data up to ~May 2026 (their own UI shows
  "No data found" after that); newer months come as PDF disclosures instead.
- RBI pages are ASP.NET postbacks; year archives need `hdnYear` + ViewState POST.
  All files download directly from rbidocs.rbi.org.in.
- RBI bank-wise UPI tables: the `NEFTView.aspx` page covers ECS/NEFT/RTGS/Mobile, not UPI.