Deprecation: LLM-based methods replace Topic method

UX improvement: Extraction history

Improvement: Select batch environment in the app

May 2024

Sensible

Getting started

Integrating

Choosing an extraction approach

SDK guides

API quickstart

Advanced prompt configuration

Getting started with layout-based extractions

Out-of-the-box extractions

Multi-document extractions

Repeating layouts

Classifying documents by type

Troubleshooting LLM extractions

Monitoring extractions

Qualifying LLM accuracy

Validating extractions

SenseML reference introduction

Code examples

Introduction

Authentication

June 2024

April 2024

March 2024

February 2024

January 2024

Tutorials

API Status

Blog

API Reference

Changelog

Try Sensible

Sign In

Overview

Zapier overview

Zapier tutorial

Advanced Zapier tutorial

Quick extraction

SenseML to spreadsheet reference

Prompt tips

Query group extraction tips

List tips

NLP table extraction tips

App guide

Color coding

Sensible walkthrough

Using fallbacks

Optimizing extraction performance

Testing before integrating configs

Troubleshooting

Extracting handwriting and OCR text

Go-live checklist

Field query object

Anchor object

Match object

Method object

Types

Methods

Checkbox

Column

Document range

Fixed table

Label

Intersection

Nearest checkbox

Paragraph

Passthrough

Regex

Region

Signature

Text Table

LLM-based methods

List

NLP table

Query group

Summarizer

Preprocessors

Deskew

Ligature

Merge lines

OCR preprocessor

Page range

Remove footer

Remove header

Rotate page

Scale

Split lines

Computed field methods

Concatenate

Constant

Mapper

Pick values

Split

Suppress output

Sections

Claims loss run example

Labeled rows example

Advanced: nested columns example

Advanced: nested table example

Advanced: Table grid example

Advanced: Zip sections

Advanced: Zip and flatten nested sections

Advanced: Transform sections data

Advanced: External anchors for sections

Config settings

Fingerprint

Verbosity

Document type settings

Fingerprint mode

OCR engine

OCR level

Advanced computed field methods

Add computed fields

Copy from sections

Copy to section

Custom computation

Get file metadata

Concepts

Accuracy measures

Anchor nuances

Bag of words

Extraction coverage

Supported file types

Field extraction order

Ligatures

Lines

Match arrays

Section nuances

SenseML

Choosing a Table method

API tutorial

Try synchronous extraction

Try asynchronous extraction from your URL

Try asynchronous extraction from a Sensible URL

Try a webhook


**Note:** Use this endpoint for testing. Use the asynchronous extraction endpoints for production.

Extract data from a local document synchronously.

To explore this endpoint, use this interactive API reference, or use one of the following options:

- For a quick "hello world" response to this endpoint, see the [API quickstart](https://sensible.mintlify.app/.app/integrations/quickstart)
- For a step-by-step tutorial about calling this endpoint, see [Try synchronous extraction](https://sensible.mintlify.app/.app/api-guides/api-tutorial/api-tutorial-sync).
- Run this endpoint in the Sensible Postman collection. [Run in Postman](https://god.gw.postman.com/run-collection/16839934-45339059-3fec-4c31-a891-9a12a3e1c22b?action=collection%2Ffork&collection-url=entityId%3D16839934-45339059-3fec-4c31-a891-9a12a3e1c22b%26entityType%3Dcollection%26workspaceId%3Ddbde09dc-b7dd-487d-a68f-20d32b008f90)

There are two options for posting the document bytes.
  1. (often preferred) specify the non-encoded document bytes as the entire request body,and specify the `Content-Type` header, for example,"application/pdf" or "image/jpeg".
     See the following for supported file formats.
  2. Base64 encode the document bytes, specify them in a body "document" field, and specify application/json for the `Content-Type` header.

For a list of  supported document file types, see [Supported file types](https://sensible.mintlify.app/.app/senseml-reference/concepts/file-types).


Extract data from a document (sync)

Extract data asynchronously from a document with the following steps:
  1. Use this endpoint to generate a Sensible URL.
  2. PUT your document at the `upload_url` returned from the previous step. Sensible extracts data from the document.
  3. To retrieve the extraction, use a webhook, or use the extraction `id` returned in the response to poll the GET documents/{id} endpoint.

For supported file size and types, see [Supported file types](https://sensible.mintlify.app/.app/senseml-reference/concepts/file-types).

For example, if your call to `/generate_upload_url` specifies the document type with a `content_type` body parameter (recommended), your first two steps are as follows:

Step 1. Generate the Sensible URL:

```curl
curl --location 'https://api.sensible.so/v0/generate_upload_url/<YOUR_DOCUMENT_TYPE>' \
--header 'Content-Type: application/json' \
--header 'Accept: application/json' \
--header 'Authorization: Bearer REDACTED' \
--data '{"content_type":"application/pdf"}'
```

Step 2. PUT the document:

```curl
curl --location --request PUT 'https://sensible-so-utility-bucket-dev-us-west-2.s3.us-west-2.amazonaws.com/REDACTED' \
--header 'Content-Type: application/pdf' \
--data 'YOUR_PATH_TO_DOCUMENT.pdf'
```

Note that in step 2:
  - you must omit an authorization header
  - the `Content-Type` header must match the `content_type` body parameter in step 1
  - the pre-signed `upload_url` doesn't support Base64 encoded documents, so you PUT the document bytes directly to the endpoint.


For a step-by-step tutorial on calling this endpoint, see
[Try asynchronous extraction from a Sensible URL](https://docs.sensible.so/docs/api-tutorial-async-2).


Extract doc at a Sensible URL

Extract data asynchronously from a document at the specified `document_url`.<br/>
For supported file size and types, see [Supported file types](https://sensible.mintlify.app/.app/senseml-reference/concepts/file-types).
Take the following steps.
1. Run this endpoint.
3. To retrieve the extraction, use a webhook, or use the extraction `id` returned in the  response to poll the GET documents/{id} endpoint.
For a step-by-step tutorial on calling this endpoint,
see [Try asynchronous extraction from your URL](https://sensible.mintlify.app/.app/api-guides/api-tutorial/api-tutorial-async-1).


Extract doc at your URL

This endpoint's behavior identical to the [Extract data from a document](ref:extract-data-from-a-document) endpoint's behavior, except that Sensible uses the specified config to extract data from the document instead of automatically choosing the best-scoring extraction in the document type.


Extract data from a document using specified config

This endpoint's behavior is identical to the [Extract doc at a Sensible URL](ref:generate-upload-url) endpoint's behavior, except that Sensible uses the specified config to extract data from the document instead of automatically choosing the best-scoring extraction in the document type.


Extract doc at a Sensible URL using specified config

This endpoint's behavior is identical to the [Extract doc at your URL](ref:extract-from-url) endpoint's behavior, except that Sensible uses the specified config to extract data from the document instead of automatically choosing the best-scoring extraction in the document type.


Extract doc at your URL using config

Use this endpoint with multiple documents that are packaged into one file (a "portfolio"). For a list of supported file types, see [Supported file types](https://sensible.mintlify.app/.app/senseml-reference/concepts/file-types).
Segments a portfolio file into the specified document types (for example, 1099, w2, and bank_statement) and then runs extractions
asynchronously for each document Sensible finds in the portfolio.  Take the following steps -
1. Use this endpoint to generate a Sensible URL.
2. PUT the document you want to extract data from at the URL, where `SENSIBLE_UPLOAD_URL` is the URL you received
from this endpoint's response. For more information about how to PUT the document, see the [generate_upload_url/{document_type}](ref:generate-upload-url) endpoint.
3. To retrieve the extraction, use a webhook, or use the extraction `id` returned in the  response to poll the GET documents/{id} endpoint.
For more about extracting from portfolios, see [Multi-document extractions](https://sensible.mintlify.app/.app/layout-based-extractions/portfolio).


Extract portfolio at a Sensible URL


Use this endpoint with multiple documents that are packaged into one file (a "portfolio"). For a list of supported file types, see [Supported file types](https://sensible.mintlify.app/.app/senseml-reference/concepts/file-types).
Segments a portfolio file at the specified `document_url` into the specified document types (for example, 1099, w2, and bank_statement)
and then runs extractions asynchronously for each document Sensible finds in the portfolio. Take the following steps.
1. Run this endpoint.
3. To retrieve the extraction, use a webhook, or use the extraction `id` returned in the  response to poll the GET documents/{id} endpoint.
For more about extracting from portfolios, see [Multi-document extractions](https://sensible.mintlify.app/.app/layout-based-extractions/portfolio).


Extract portfolio at your URL

Use this endpoint in conjunction with asynchronous extraction requests to retrieve your results.
You can also use this endpoint to retrieve the results for documents extractions from the synchronous /extract endpoint.
To poll extraction status, check the `status` field in this endpoint's response.
When the extraction completes, the returned status is `COMPLETE` and the response includes results in the
`parsed_document` field.  For fields in the extraction for which Sensible couldn't find a value, Sensible returns null.


Retrieve extraction by ID

Use this endpoint to get a filtered list of past extractions.
This endpoint returns a summary for each extraction, listed in reverse chronological order.
To get details about an extraction, use the [Retrieve extraction by ID](ref:retrieving-results) endpoint.
This endpoint uses keyset pagination to retrieve the next page of results.
By default it returns a first page of 20 extractions and an opaque `continuation_token` that you can pass in the next request to get the next page of results, until the endpoint returns `continuation_token` to indicate the last page.
Use the `limit` parameter to configure page size.


List extractions

Returns daily extraction coverage statistics per config. Sensible returns coverage for each config that was used for at least one extraction performed in the production environment in the specified time period. For more information about coverage, see [Monitoring extractions](metrics).


Get extraction statistics

You can use this endpoint to get Excel files from documents, for example from PDFs. In more detail, this endpoint converts your JSON document extraction to an Excel spreadsheet.
To compile multiple documents into one Excel file, specify the IDs of their recent extractions in the request separated by commas, for example,
`/generate_excel/867514cc-fce7-40eb-8e9d-e6ec48cdac34,5093c65f-05bd-46a3-8df7-da3ed00f6d35`.
For the best compiled spreadsheet results, configure your SenseML so that the documents output identically named fields.
For more information about the conversion process, see [SenseML to spreadsheet reference](https://sensible.mintlify.app/.app/integrations/quick-extraction/excel-reference).

For portfolio extractions, Sensible returns an Excel file containing fields for all the documents it finds in the PDF. For more information, see [Multi-document spreadsheet](https://sensible.mintlify.app/.app/integrations/quick-extraction/excel-reference#multi-document-spreadsheet).

For a list of document file types that Sensible can extract data from, see [Supported file types](https://sensible.mintlify.app/.app/senseml-reference/concepts/file-types).
Call this endpoint after an extraction completes. For more information about checking extraction status,
see the `GET /documents/{id}` endpoint.


Get Excel extraction

You can use this endpoint to get CSV files from documents, for example, from PDFs. In more detail, this endpoint converts your JSON document extraction to a comma-separated values.
To compile multiple documents into one CSV file, specify the IDs of their recent extractions in the request separated by commas, for example,
`/generate_csv/867514cc-fce7-40eb-8e9d-e6ec48cdac34,5093c65f-05bd-46a3-8df7-da3ed00f6d35`.
For the best compiled spreadsheet results, configure your SenseML so that the documents output identically named fields.
For more information about the conversion process, see [SenseML to spreadsheet reference](https://sensible.mintlify.app/.app/integrations/quick-extraction/excel-reference).
For a list of document file types that Sensible can extract data from, see [Supported file types](https://sensible.mintlify.app/.app/senseml-reference/concepts/file-types).
Call this endpoint after an extraction completes. For more information about checking extraction status,
see the `GET /documents/{id}` endpoint.


Get CSV extraction

Score a document's similarity to each document type you defined in your Sensible account and to each reference document in the highest-scoring type.
To retrieve the scores, poll the `download_link` in this endpoint's response until it returns a non-error response.
This endpoint is asynchronous. For more information about scores, expand the 200 response in the synchronous [classification](ref:classify-document-sync) endpoint.

Use this endpoint:

 - In an extraction workflow. For example, determine which documents to extract prior to calling a Sensible extraction endpoint.
 - Outside an extraction workflow. For example, to determine where to route each document or to label each document in a system of record.

To post the document bytes, specify the non-encoded document bytes as the entire request body,and specify the `Content-Type` header, for example,"application/pdf" or "image/jpeg".

For supported file size and types, see [Supported file types](https://sensible.mintlify.app/.app/senseml-reference/concepts/file-types).


Classify document by type

**Note:** Use this Classify endpoint for testing. Use the asynchronous Classify endpoint for production.

Score a document's similarity to each document type you defined in your Sensible account. Get scores for the document's similarity to document types and to their reference documents.

Use this endpoint:

 - In an extraction workflow. For example, determine which documents to extract prior to calling a Sensible extraction endpoint.
 - Outside an extraction workflow. For example, determine where to route each document or to label each document in a system of record.

To post the document bytes, specify the non-encoded document bytes as the entire request body,and specify the `Content-Type` header, for example,"application/pdf" or "image/jpeg".

For supported file size and types, see [Supported file types](https://sensible.mintlify.app/.app/senseml-reference/concepts/file-types).


Classify document by type (sync)

List all document types for this account.

List document types for this account

Create document type

Find the document type id using the `/document_types` endpoint.

Get document type

Update an existing document type with new information. For example, use this endpoint to add validations:

```curl
curl --location --request PUT 'https://api.sensible.so/v0/document_types/<TYPE_ID>' \
--header 'Authorization: Bearer YOUR_API_KEY' \
--header 'Content-Type: application/json' \
--data-raw '{"schema":{"validations":[{"description":"example validation to test broker email format","condition":{"match":[{"var":"broker\\.email.value"},"^\\S+\\@\\S+$"]},"severity":"warning","fields":["test"]}]}} '
```


Update document type

Delete a document type and everything it contains, for example configurations and reference PDfs,
but not its extraction history displayed in the Sensible app.


Delete document type

List configurations in a document type

Pass the configuration as stringified JSON.

Create configuration in a document type

Get configuration

Replace a published or draft version of the configuration.

Update configuration

Delete configuration

List versions for a configuration

Get a configuration as stringified JSON by version id.

Get configuration by version

To publish to an environment instead of as the current draft, the configuration must be valid according to this [schema](https://schema.sensible.so/configuration.schema.json).

Publish configuration to an environment

To delete a draft, specify a version name in the `version` parameter. To unpublish a configuration, enter the publication environment name in the `version` parameter, for example, `development`.

Delete draft or unpublish configuration

List all reference documents in a document type

Specify document metadata in the request, and get back an `upload_url` at which to put the PDF, for example with `curl -T ./sample.pdf`.

Changelog

​Deprecation: LLM-based methods replace Topic method

​UX improvement: Extraction history

​Improvement: Select batch environment in the app

Deprecation: LLM-based methods replace Topic method

UX improvement: Extraction history

Improvement: Select batch environment in the app