Vision

Three things a phone can do with a photo, entirely on-device: cut the subject out, describe what is in it, and read the text in it.

Overview

expo-ai-kit uses each platform's own vision engine: Apple's Vision framework on iOS and ML Kit on Android. Nothing is uploaded, images are read from the device and the results come back as plain data.

• removeBackground(): a cutout of the subject with a transparent background, saved as a PNG file
• labelImage(): ranked labels describing the image (“Dog”, “Beach”, “Food”, …)
• recognizeText(): the text in the image, with normalized bounds for every block and line
• The same lifecycle style as the rest of the library: explicit availability, one preparation call, typed errors

Enable vision (Android)

iOS needs no configuration, the Vision framework ships with the OS. Android is off by default because it adds the ML Kit clients and a bundled label model to the APK. Turn it on in your app config and make a new native build (dev client or EAS, not an OTA update):

{
  "expo": {
    "plugins": [["expo-ai-kit", { "vision": true }]]
  }
}

No permissions are added: your app reads image files it already has access to (for example from expo-image-picker or the camera). Without the flag, Android vision APIs throw a typed VISION_NOT_ENABLED error and the app pays zero size cost.

Availability & preparation

Each feature reports its own status. On Android, subject segmentation and text recognition are Google Play services models that download once; prepareVision() is the only call that downloads them. Image labeling ships inside the app and works offline immediately. On iOS every model is part of the OS, so preparation resolves at once.

import { getVisionAvailability, prepareVision } from 'expo-ai-kit';

const availability = await getVisionAvailability();
// availability.backgroundRemoval / imageLabeling / textRecognition, each one of:
//   { status: 'available' }
//   { status: 'downloadable' | 'downloading' }, Android model not installed yet
//   { status: 'unavailable', reason: 'platform' | 'os-version' | 'device' | 'not-enabled' }

if (availability.backgroundRemoval.status === 'downloadable') {
  await prepareVision({
    features: ['background-removal'],
    onProgress: (progress) => console.log(progress), // 0..1
  });
}

// For OCR in non-Latin scripts, name the languages so Android fetches those models:
await prepareVision({ features: ['text-recognition'], languages: ['ja', 'zh-Hans'] });

Background removal

Lift the subject out of a photo. The cutout is written to the app's cache directory and returned as a file:// URI, pixel data never crosses the bridge, along with where the subject sits in the source image.

import { removeBackground } from 'expo-ai-kit';

const cutout = await removeBackground(
  { uri: photo.uri },
  {
    trim: true,           // crop to the subject (default); false keeps the full frame
    format: 'png',        // 'png' keeps transparency (default); 'jpeg' flattens onto white
    quality: 0.9,         // JPEG only
    maxPixels: 6_000_000, // downscale larger images first (default)
    subject: { x, y },    // optional: keep only the subject under this normalized point
    mask: true,           // optional: also write the mask as a grayscale PNG
  }
);

<Image source={{ uri: cutout.uri }} style={{ width: cutout.width, height: cutout.height }} />

cutout.maskUri;            // grayscale mask PNG, white = subject (when mask: true)
cutout.bounds;             // subject bounds in the source, normalized 0–1
cutout.pixelBounds;        // the same in source pixels
cutout.foregroundCoverage; // fraction of the image that is subject
cutout.centroid;           // subject centre, normalized
cutout.instanceCount;      // how many subjects the engine found in the image

By default every subject the engine finds is kept. Pass subject with a point normalized to the source image (origin top-left, 0 to 1), for example where the user tapped, to keep only the subject under it; both engines segment subjects individually. Pass mask: true to also get the mask the engine used as an 8-bit grayscale PNG the same size as the output, for your own compositing, feathering, or editing. The files live in the cache, so move or copy them if you need to keep them. An image with no clear subject, or no subject under the given point, throws NO_SUBJECT_FOUND.

Image labels

Describe what is in an image. Labels come back sorted by confidence, highest first; the default keeps the top 10 above 50% confidence.

import { labelImage } from 'expo-ai-kit';

const labels = await labelImage({ uri: photo.uri }, { maxResults: 5, minConfidence: 0.6 });
// [{ label: 'Dog', confidence: 0.97 }, { label: 'Pet', confidence: 0.88 }, …]

// maxResults: 0 returns every label above minConfidence.

iOS uses Vision's image classifier (about 1,300 labels); Android uses ML Kit Image Labeling (about 400 labels). The vocabularies and formats differ, iOS returns Vision's lowercase identifiers such as adult or consumer_electronics, Android returns ML Kit's capitalized words such as Dog or Pet, so treat labels as descriptive strings for display and search, not as a shared taxonomy to switch on.

Text recognition (OCR)

Read the text in a photo, screenshot, or document. You get the full text plus each block and line with normalized bounds (origin top-left, 0–1), so you can draw overlays or pick regions.

import { recognizeText } from 'expo-ai-kit';

const result = await recognizeText(
  { uri: photo.uri },
  {
    languages: ['en', 'de'],  // optional, iOS auto-detects; Android reads Latin by default
    minTextHeight: 0.02,      // ignore text shorter than 2% of the image height
    // iOS only: recognitionLevel: 'accurate' | 'fast', usesLanguageCorrection, customWords
  }
);

result.text;    // all text, blocks joined with newlines
result.blocks;  // [{ text, bounds, lines: [{ text, bounds, confidence?, language? }] }]

Android's languages selects ML Kit script models, Latin, Chinese, Japanese, Korean, and Devanagari, each a one-time Play services download made by prepareVision(). iOS asks the Vision framework. Ask the running device what it can read:

import { getSupportedTextRecognitionLanguages } from 'expo-ai-kit';

const languages = await getSupportedTextRecognitionLanguages();
// iOS: ['de-DE', 'en-US', 'fr-FR', 'ja-JP', 'zh-Hans', …]   Android: ['en', 'de', 'zh', 'ja', 'ko', 'hi', …]

Image input

Every vision call takes { uri }: a file:// URI or absolute path (Android also accepts content://). Photos from expo-image-picker, expo-camera, and the file system work directly. EXIF orientation is applied before processing, so results match what the user sees. Large images are downscaled to a per-feature pixel budget before the model runs; the returned coordinates are normalized so they stay correct for the original.

Platform notes

• iOS: Vision framework. Background removal needs iOS 17+. The iOS Simulator cannot run Vision's neural requests, so background removal and image labeling report { status: 'unavailable', reason: 'device' } there and need a physical device; text recognition works in the Simulator too.
• Android: ML Kit. Background removal and OCR need Google Play services (the models are Play services modules); image labeling is bundled and works without it. Devices without Play services report reason: 'device' for those two.
• Vision calls are independent of the text and speech guards, they run alongside a generation or a transcription without tripping INFERENCE_BUSY or SPEECH_BUSY.
• Vision has no AI SDK model type; use these core functions directly.

Errors

Vision failures throw ModelError with a typed .code: VISION_NOT_ENABLED, IMAGE_DECODE_FAILED, NO_SUBJECT_FOUND, VISION_FAILED, plus the shared DEVICE_NOT_SUPPORTED, MODEL_NOT_DOWNLOADED, LANGUAGE_NOT_SUPPORTED, and DOWNLOAD_FAILED. See Troubleshooting for the full table.