1. Overview & Business Case
A retail and commercial bank serving over 8 million customers across personal banking, mortgages, small business lending, and wealth management generates an enormous volume of unstructured text every day — customer service chats, call transcripts, loan documents, complaint letters, KYC filings, and compliance correspondence. Historically, almost none of this text was systematically analysed; it was read once, actioned manually, and archived.
Banks sit on some of the richest unstructured text data of any industry — and some of the strictest regulatory constraints on how AI can act on it. The opportunity isn't just automating reading; it's building NLP infrastructure that understands intent, sentiment, and risk in text while staying fully auditable, explainable, and compliant with banking regulation.
Business Problems This System Solves
- Slow, inconsistent customer service: Customers waited an average of 6+ minutes for a live agent for routine queries (balance, transaction disputes, card issues) that don't require human judgment.
- Invisible sentiment risk: Negative sentiment in call transcripts, complaint emails, and chat logs went undetected until it escalated into formal complaints or regulatory referrals.
- Manual document processing bottlenecks: Loan officers spent hours manually extracting data from pay stubs, bank statements, and identity documents — a process prone to error and inconsistent turnaround time.
- Compliance correspondence backlog: Regulatory and legal correspondence required careful, consistent classification and routing — a process that didn't scale with growing correspondence volume.
2. System Architecture
The platform follows the same layered microservices philosophy used across JoNoleecy AI implementations — a clean separation between presentation, business services, AI/LLM inference, agent orchestration, and data, with banking-specific compliance and audit layers woven through every tier.
Technology Stack
| Layer | Technology | Rationale |
|---|---|---|
| Backend | Java 17 + Spring Boot 3.2 + Spring Security | Enterprise-grade reliability and battle-tested security posture demanded by banking regulators |
| Frontend | React 18 + TypeScript | Real-time chat widget and agent-assist console with WebSocket-driven live updates |
| Primary Database | MySQL 8.0 (AWS RDS Multi-AZ, encrypted at rest) | Conversation logs, case records, document metadata — full audit trail, point-in-time recovery |
| Document Store | AWS S3 (KMS-encrypted, versioned) | Original uploaded documents retained for compliance and audit, never exposed to LLMs directly |
| Event Streaming | Apache Kafka (MSK, private subnet) | Conversation events, sentiment alerts, document processing events — decoupled, auditable flow |
| Cache | Redis 7 Cluster | Conversation context cache, intent classification cache — sub-millisecond response |
| Vector DB | pgvector (on RDS PostgreSQL) | Document and FAQ embeddings for semantic chatbot retrieval, kept within the bank's compliance boundary |
| LLM Primary | Claude 3.5 Sonnet (via AWS Bedrock, in-VPC) | Strong reasoning and tone control for customer-facing banking conversations; data stays in-region |
| LLM Secondary | GPT-4o (Azure OpenAI, private endpoint) | Document summarisation and complex multi-document reasoning tasks |
| Sentiment Model | FinBERT (fine-tuned, self-hosted) | Domain-specific financial sentiment model — outperforms general sentiment models on banking language |
| Document OCR | AWS Textract + custom parsers | High-accuracy extraction from pay stubs, statements, and identity documents |
| ML Platform | AWS SageMaker + MLflow | Managed training, experiment tracking, model registry for sentiment and classification models |
Architecture Principles
- No raw PII reaches third-party LLM APIs: A dedicated redaction layer strips or tokenises personally identifiable and account information before any prompt leaves the bank's private network boundary.
- Every AI decision is logged and explainable: Intent classifications, sentiment scores, and routing decisions are written to an immutable audit table with the full reasoning context.
- Chatbot escalates rather than guesses: Any query touching account-specific financial advice, fraud, or hardship is routed to a human agent — the AI never fabricates financial guidance.
- Human-in-the-loop for all customer-impacting actions: The AI drafts, classifies, and recommends; a human banker or compliance officer approves anything that changes an account or commits the bank.
3. Data Model — MySQL Schema Design
The core schema centres on Conversations and Documents as first-class entities, with every NLP output — intent, sentiment, extracted fields — stored as a structured, queryable record rather than buried in free text.
Core Entities: Conversations & Messages
Document Processing & Compliance Tables
Event-Driven Architecture with Kafka
4. Backend — Spring Boot Conversation Service
The backend centres on a Conversation Service that mediates every customer interaction — applying PII redaction before any text reaches an LLM, orchestrating intent classification and response generation, and enforcing escalation rules before a reply is ever sent to a customer.
Conversation Service — Core Logic
Whether to escalate to a human is never left to the LLM's judgment alone. A fixed list of intents — fraud reports, hardship/financial difficulty disclosures, account closures, complaints, and any sentiment below a hard threshold — is enforced in Java code before the LLM is even invoked for response generation. This guarantees regulatory-sensitive conversations never get an AI-only resolution.
5. AI & LLM Layer
The platform separates a domain-tuned sentiment model (FinBERT, self-hosted for full data control) from a general-purpose LLM responsible for understanding conversational intent and drafting natural-language responses — never the reverse.
LLM Selection: Claude 3.5 Sonnet via AWS Bedrock
| Criterion | Why It Matters Here | Model Choice |
|---|---|---|
| Data residency | Customer conversation data cannot leave the bank's approved cloud boundary | AWS Bedrock private endpoint — data never leaves the bank's AWS account/region |
| Tone & restraint | Banking conversations require careful, accurate, non-speculative language — especially around financial advice | Claude 3.5 Sonnet — strong instruction-following and refusal behaviour on out-of-scope requests |
| Document reasoning | Multi-page loan documents need cross-referencing and summarisation | GPT-4o (private endpoint) for complex document Q&A tasks |
| Domain sentiment accuracy | General sentiment models misread banking-specific language (e.g., "frozen account" as neutral, not negative) | FinBERT — fine-tuned specifically on financial services text |
Chatbot Response Generation — Prompt Design
LLM Use Cases Across the Platform
- Customer chatbot responses: real-time drafting of replies for routine, low-risk banking queries, with mandatory handoff for anything sensitive.
- Call transcript summarisation: condensing 10–20 minute call transcripts into structured summaries for the CRM, reducing post-call wrap-up time for agents.
- Document field extraction reasoning: resolving ambiguous OCR output (e.g., distinguishing gross vs. net income on a pay stub) using contextual document understanding.
- Compliance case classification rationale: generating a plain-language explanation of why a piece of correspondence was routed to a specific department, for the human reviewer's reference.
6. Multi-Agent Architecture
Six specialised agents collaborate under a top-level orchestrator, each scoped to a narrow, auditable responsibility — deliberately avoiding any single agent with broad discretion over customer-facing or compliance-sensitive decisions.
Intent classification, sentiment scoring, document extraction, and compliance routing each demand different data access, different risk tolerance, and different audit requirements. Narrow, single-purpose agents are dramatically easier to test, certify, and explain to regulators than one large agent attempting everything.
Agent Roster
Conversation Orchestrator Agent
Top-level agent receiving every inbound message. Sequences the redaction, classification, and response pipeline, and decides whether a specialist agent or a direct handoff is required. Implemented with LangGraph4j for stateful, auditable multi-step orchestration.
Intent Classification Agent
Classifies each inbound message into one of ~40 predefined banking intents (balance inquiry, dispute, card replacement, hardship request, complaint, etc.) using a fine-tuned classifier backed by an LLM fallback for ambiguous cases.
Sentiment & Escalation Agent
Scores sentiment on every message using FinBERT, tracks sentiment trajectory across a conversation, and triggers escalation when sentiment crosses critical thresholds or shows a sustained negative trend — even if individual messages aren't extreme.
Document Extraction Agent
Orchestrates OCR (AWS Textract) plus LLM-assisted field resolution for uploaded loan documents, pay stubs, and identity documents — flagging any extraction below a confidence threshold for mandatory human review.
Compliance Routing Agent
Classifies regulatory correspondence and formal complaints into the correct internal category and routes to the right department queue, attaching a full reasoning trail for the human compliance officer who ultimately reviews and actions the case.
Compliance Guard Agent
An independent agent that re-reviews every AI-generated customer-facing response specifically for regulatory tone, prohibited financial advice language, and fair-lending considerations — structurally separate from the agent that generated the content.
7. MCP — Model Context Protocol Integration
MCP gives the agent layer structured, permissioned access to account data, document storage, and the compliance case system — without ever embedding raw SQL or direct data access inside an LLM prompt.
In a banking context, uncontrolled data access by an LLM agent is a regulatory non-starter. MCP enforces a typed, logged contract: each tool exposes only the specific, minimal data an agent needs for its task, and every call is recorded with the requesting agent, the customer context, and a timestamp.
MCP Server Design — Banking NLP Tools
| MCP Tool Name | Description | Returns |
|---|---|---|
get_account_summary_redacted | Returns a PII-redacted account summary (balance range, product type, recent activity flag) — never raw numbers | AccountSummary (JSON) |
get_conversation_history | Returns the recent message history for a conversation, already redacted | List<Message> |
extract_document_fields | Triggers OCR and field extraction for an uploaded document, returns structured fields with confidence scores | DocumentExtraction |
classify_compliance_case | Runs the compliance classification model on correspondence text and returns category + routing recommendation | ComplianceClassification |
escalate_to_human_agent | Write-tool: hands off a conversation to a live agent queue with full context | EscalationResult |
8. React Frontend
The frontend has two primary surfaces: a customer-facing Chat Widget embedded in online banking, and an internal Agent Assist Console giving human agents live sentiment and intent context the moment a conversation is escalated to them.
The Suggested Response Panel offers draft language an agent can use, edit, or ignore entirely — it never auto-sends on an escalated conversation. Once a human is in the loop, every word sent to the customer is reviewed and sent by that human.
9. NLP Pipeline Detail
Four distinct NLP capabilities power the platform, each with its own model, evaluation criteria, and failure-handling approach.
Capability 1: Intent Classification
Capability 2: Sentiment Analysis (FinBERT)
Capability 3: Document Processing & Field Extraction
Capability 4: Conversational Chatbot
The chatbot uses a retrieval-augmented generation (RAG) pattern: customer questions are matched against an embedded knowledge base of approved banking FAQs and product information via pgvector similarity search, and the LLM is instructed to ground its answer strictly in retrieved content — reducing hallucination risk on factual banking policy questions.
10. Key System Flows
Flow 1: Routine Chatbot Resolution
Message Received & Redacted
Customer asks "What's my current balance?" via the chat widget. PII redaction service tokenises any account-identifying text before it leaves the secure boundary.
Intent Classified
DistilBERT classifier identifies BALANCE_INQUIRY with 0.97 confidence — well within the auto-resolution threshold, not on the escalation list.
Response Generated & Sent
Claude 3.5 Sonnet drafts a response grounded in the redacted account summary fetched via MCP. Response cites the source and is sent directly — no human needed.
Conversation Logged
Full interaction, intent, and confidence score written to the audit table. No escalation event fired.
Flow 2: Sentiment-Driven Escalation
Declining Sentiment Detected
Customer's third consecutive message in a dispute conversation shows sentiment dropping from -0.2 to -0.6. The Sentiment & Escalation Agent fires before any single message crosses the critical threshold alone.
Hard Escalation Rule Triggered
Conversation Orchestrator halts AI auto-response generation and publishes an escalation event to Kafka with full conversation context.
Agent Assist Console Populated
A human agent's console immediately shows the sentiment trend chart, detected intent, and an AI-drafted (but unsent) suggested response for the agent to review and adapt.
Human Takes Over
Agent reviews context in seconds rather than re-reading the full transcript, and responds directly — the AI's role ends at providing context and a draft, never sending on its own.
11. Compliance & Guardrails
Banking AI carries regulatory weight that most other industries don't — fair lending rules, complaint handling regulations, and data protection law all constrain what this system is permitted to do autonomously.
Defence-in-Depth: Four Layers of Guardrails
PII Redaction Before LLM Access
No raw account number, SSN, or card number ever reaches a third-party LLM API — redaction happens in Java code before any prompt is constructed.
Hard-Coded Escalation Rules
A fixed, code-level list of intents and sentiment thresholds forces human handoff — this is never left to LLM discretion, regardless of how the prompt is engineered.
Compliance Guard Agent Review
An independent agent re-reads every AI-generated customer-facing response specifically for prohibited financial advice language and fair-lending tone — separate from the agent that drafted it.
Immutable Audit Trail
Every classification, sentiment score, and routing decision is logged with full reasoning context — satisfying both internal QA and regulatory examination.
Compliance Rule Examples
| Check | Trigger Condition | System Action |
|---|---|---|
| Financial Advice Detection | Response draft contains personalised investment or lending recommendation language | Response blocked by Compliance Guard Agent before send; rewritten or escalated |
| Fraud Mention | Customer message contains fraud-related keywords or patterns | Immediate hard escalation, bypassing all AI auto-response logic |
| Hardship Disclosure | Message indicates financial difficulty or hardship | Routed to a specially trained human team, never AI-resolved |
| Document Confidence Below Threshold | Field extraction confidence < 0.75 on any loan-relevant document | Document flagged for mandatory human verification before use in underwriting |
| Sentiment Deterioration Trend | 3+ consecutive messages with declining sentiment | Auto-escalation even without a single critical-threshold message |
12. Deployment & MLOps
The platform runs entirely within the bank's private AWS VPC on EKS, with all LLM calls routed through private endpoints (Bedrock and Azure OpenAI private link) so no conversation data traverses the public internet.
| Capability | Implementation | Scale Target |
|---|---|---|
| Real-time chat response | Spring WebFlux + Kubernetes HPA | <2s p99 response time |
| Sentiment scoring | Self-hosted FinBERT on SageMaker endpoints | <200ms per message |
| Document extraction (batch) | AWS Textract + Spring Batch | Loan document set in <5 minutes |
| Model retraining (sentiment) | SageMaker Training Job, monthly | Full retrain in <3 hours |
| Model monitoring | Evidently AI + Splunk compliance dashboard | Daily drift checks, real-time anomaly alerts |
| LLM cost management | Prompt caching, model routing by task complexity | Cost tracked per conversation, budget alerts |
| Audit logging | Immutable append-only log to Splunk + S3 Glacier | 7-year regulatory retention |
13. Outcomes & Metrics
Measured over a 6-month production period across the bank's digital channels, compared against the prior human-only baseline:
Beyond the headline numbers, human agents consistently reported that the Agent Assist Console's instant sentiment and intent context — rather than having to re-read a full transcript on every escalation — was the single most valued feature, meaningfully reducing time-to-resolution on already-frustrated customer interactions.