In high-throughput clinical workflows, manual data entry is a major source of friction and administrative error. When a patient registers in a healthcare portal, typing long alphanumeric policy, group, and claim numbers from an insurance card is tedious and prone to typos. Such typos can lead to claim rejections, delayed authorizations, and administrative overhead.

To solve this, modern healthcare portals leverage Vision LLMs to extract insurance data automatically from uploaded images. However, in a HIPAA-regulated environment, you cannot simply send patient medical images directly from the browser to public AI endpoints.

This article details how to build a secure, HIPAA-Compliant Vision AI Extraction Pipeline using the "Proxied AI" pattern. We'll implement a custom AWS Medplum FHIR Operation ($aws-bedrock) in Node.js, process images via Anthropic Claude 3.5 Sonnet on AWS Bedrock, resolve complex file storage endpoints, and map extracted data back to a React frontend.

🏗️ The Server-Side "Proxied AI" Architecture

Rather than having the frontend call the AI services directly, all interactions with AWS Bedrock are proxied through a secure, server-side FHIR operation. This keeps AWS credentials out of the browser, guarantees patient-level authentication, and ensures all images are validated before being processed:

🛠️ The Backend: Implementing Custom $aws-bedrock Operation

In Medplum, custom operations are registered as server-side route plugins. When a client requests $aws-bedrock on a Media resource, the backend retrieves the image, reads the file buffer directly from S3 (bypassing the browser's Content Security Policy constraints), invokes Bedrock, and returns a verified JSON object.

Below is the production-ready Node.js/TypeScript backend operation handler:

// bedrock.ts (AWS Bedrock-FHIR Operation Handler)
import { MedplumServer, getBinaryStorage } from '@medplum/server';
import { BedrockRuntimeClient, InvokeModelCommand } from '@aws-sdk/client-bedrock-runtime';
import { Media, Binary } from '@medplum/fhir';
import Jimp from 'jimp';
import { logger } from './logger.js';

const bedrockClient = new BedrockRuntimeClient({ region: 'us-east-1' });

export async function awsBedrockHandler(req: any, res: any): Promise<void> {
  const { id } = req.params;
  const server = MedplumServer.getInstance();

  try {
    // 1. Retrieve the FHIR Media Resource and enforce client compartment check
    const media = await server.readResource<Media>('Media', id);
    if (!media.content?.url) {
      res.status(400).json({ error: 'Media resource is missing content reference.' });
      return;
    }

    // 2. Fetch image buffer from S3 or internal storage
    const buffer = await resolveImageBuffer(media.content.url);
    const contentType = media.content.contentType || 'image/jpeg';

    // 3. Optimize Image Size (AWS Bedrock has a 2MB limit on image input payloads)
    const optimizedBuffer = await optimizeImageSize(buffer);

    // 4. Invoke Claude 3.5 Sonnet on AWS Bedrock
    const extractedData = await invokeClaudeVision(optimizedBuffer, contentType);

    // 5. Send structured JSON back in the FHIR response
    res.status(200).json(extractedData);
  } catch (err: any) {
    logger.error({ err, id }, 'Error executing $aws-bedrock operation.');
    res.status(500).json({ error: 'Failed to process clinical image through Bedrock.' });
  }
}

async function resolveImageBuffer(url: string): Promise<Buffer> {
  // If reference is internal FHIR Binary
  if (url.startsWith('Binary/')) {
    const binaryId = url.split('/')[1];
    return await getBinaryStorage().readBinary(binaryId);
  }

  // If reference is an absolute S3 URI
  if (url.startsWith('s3://')) {
    const s3Path = url.replace('s3://', '');
    const [bucket, ...keyParts] = s3Path.split('/');
    return await getBinaryStorage().readFileFromBucketKey(bucket, keyParts.join('/'));
  }

  throw new Error(`Unsupported image storage URL format: ${url}`);
}

async function optimizeImageSize(buffer: Buffer): Promise<Buffer> {
  if (buffer.length < 2 * 1024 * 1024) return buffer; // Under 2MB; no processing needed

  // Downscale and compress high-resolution card uploads
  logger.info('Image size exceeds 2MB limit. Resizing via Jimp...');
  const image = await Jimp.read(buffer);
  return await image
    .resize(1024, Jimp.AUTO) // Cap width to 1024px
    .quality(80) // Re-encode to 80% JPEG quality
    .getBufferAsync(Jimp.MIME_JPEG);
}

async function invokeClaudeVision(buffer: Buffer, contentType: string): Promise<any> {
  const modelId = 'us.anthropic.claude-3-5-sonnet-20241022-v2:0';
  const prompt = `Analyze this insurance card image. Extract all relevant information and format it as a JSON object with the following keys:
  - companyName
  - policyNumber
  - groupNumber
  - planNumber
  - claimNumber
  - insuredFirstName
  - insuredLastName
  
  Also provide a field 'rawText' containing all plain text found on the card.
  Return ONLY the JSON object. Do not include markdown wraps or conversational preambles.`;

  const payload = {
    anthropic_version: 'bedrock-2023-05-31',
    max_tokens: 1000,
    messages: [
      {
        role: 'user',
        content: [
          {
            type: 'image',
            source: {
              type: 'base64',
              media_type: contentType === 'image/png' ? 'image/png' : 'image/jpeg',
              data: buffer.toString('base64')
            }
          },
          {
            type: 'text',
            text: prompt
          }
        ]
      }
    ]
  };

  const command = new InvokeModelCommand({
    modelId: modelId,
    body: JSON.stringify(payload),
    contentType: 'application/json'
  });

  const response = await bedrockClient.send(command);
  const textResponse = new TextDecoder().decode(response.body);
  const result = JSON.parse(textResponse);
  const rawText = result.content[0].text;

  // Robust parsing: extract the JSON object from LLM chatter using regex
  const jsonMatch = rawText.match(/\{[\s\S]*\}/);
  if (!jsonMatch) {
    throw new Error('LLM did not return a valid JSON object structure.');
  }

  return JSON.parse(jsonMatch[0]);
}

🛡️ The Frontend: React Portal Integration and Auditing

Once the $aws-bedrock custom FHIR endpoint is active, we trigger the processing in our React patient portal.

For maximum clinical integrity, we implement an "Invisible Storage" Auditing Pattern. The full plain text output of the card (rawText) is crucial for administrative review and syncs to backend FileMaker databases, but it is too messy and confusing for patients to read. The portal stores the front/back OCR data silently in the Mantine form state without rendering them as textareas, and includes them explicitly in the final submission payload.

// PatientIntakeQuestionnairePage.tsx (React Frontend Integration Snippet)
import React, { useState } from 'react';
import { useForm } from '@mantine/form';
import { Button, notifications } from '@mantine/core';
import { useMedplum } from '@medplum/react';
import { IconSparkles } from '@tabler/icons-react';
import { logger } from './logger.js';

export function InsuranceQuestionnaireComponent() {
  const medplum = useMedplum();
  const [isExtracting, setIsExtracting] = useState(false);

  const form = useForm({
    initialValues: {
      primaryInsurance: {
        companyName: '',
        policyNumber: '',
        groupNumber: '',
        cardFrontMediaId: '', // Internal FHIR Media ID references
        cardFrontDetails: '', // Hidden OCR audit storage
        cardBackDetails: ''
      }
    }
  });

  const handleFillFromImages = async () => {
    const frontId = form.values.primaryInsurance.cardFrontMediaId;
    if (!frontId) {
      notifications.show({ color: 'red', message: 'Please upload the front card image first.' });
      return;
    }

    setIsExtracting(true);

    try {
      // Post directly to the proxied server-side FHIR operation
      const result = await medplum.post(
        medplum.fhirUrl('Media', frontId, '$aws-bedrock'),
        {}
      );

      // Verify the AI returned structured content
      if (!result.policyNumber && !result.companyName) {
        throw new Error('Could not parse clinical text from image.');
      }

      // Map structured output safely to form state
      form.setFieldValue('primaryInsurance.companyName', result.companyName || '');
      form.setFieldValue('primaryInsurance.policyNumber', result.policyNumber || '');
      form.setFieldValue('primaryInsurance.groupNumber', result.groupNumber || '');
      
      // Save full OCR raw text for administrative compliance audit trails
      form.setFieldValue('primaryInsurance.cardFrontDetails', result.rawText || '');

      notifications.show({
        color: 'green',
        message: 'Successfully populated insurance details from card photo!'
      });
    } catch (err) {
      logger.error({ err }, '[Fill from Images] Processing failed.');
      notifications.show({
        color: 'red',
        message: 'Vision AI failed to read your card. Please verify details manually.'
      });
    } finally {
      setIsExtracting(false);
    }
  };

  return (
    <div className="insurance-upload-section">
      <Button
        leftSection={<IconSparkles size={16} />}
        onClick={handleFillFromImages}
        loading={isExtracting}
        disabled={!form.values.primaryInsurance.cardFrontMediaId}
      >
        Autofill from Uploaded Card
      </Button>
    </div>
  );
}

📈 Summary of Benefits

Connecting AWS Bedrock Claude 3.5 Sonnet to server-side Medplum operations offers critical security and operational dividends:

1. HIPAA & CSP Isolation: Frontend codes never load AWS credentials. File streaming, CSP headers, and access logic are entirely encapsulated within safe backend operation compartments.
2. Absolute Data Integrity: High-res smartphone camera uploads often exceed 10MB. Jimp compression limits payloads to 2MB, reducing Bedrock request drops and keeping execution times under 15 seconds.
3. Preamble Immunity: Using regex JSON parsing (/\{[\s\S]*\}/) ensures that conversational chatter from the LLM never causes JSON parsing failures.
4. Clinical Auditing Trails: Capturing the raw OCR output silently and syncing it to FileMaker preserves a pristine audit trail, enabling clinicians to verify AI extractions directly from layout screens.

By building a server-side "Proxied AI" handler inside a serverless Medplum framework, you create a state-of-the-art insurance processing engine that is highly secure, exceptionally fast, and fully compliant with healthcare data security laws.