Developers

Build an Atlas connector

Extend the @futurity/atlas-connector SDK, scaffold with bun create atlas-connector, prove it with atlas-conform, and connect it as a source.

A connector is a small HTTP service you host that answers Atlas's queries over data Atlas cannot read first-hand: a business system that only speaks its own API, a database behind a VPN, a warehouse with no first-party driver. Atlas discovers its tables, measures its keys, and plans queries over it next to every other source in the same Atlas. The @futurity/atlas-connector SDK owns the wire protocol, the server, bearer auth, timeouts, NDJSON streaming, and the error envelope. You write the part only you can: describing your data and fetching it.

One connector, many tenants

A connector is written once and hosted once, by whoever knows the upstream system. Every client of that integrator connects the same hosted connector into their own Atlas, typing their own upstream credentials into the connect form. Those credentials travel on every request Atlas sends, and the connector holds nothing between calls. The host never stores a tenant's secret, an added tenant is a form someone fills in, and a leaked connector process has no customer secret to leak.

                       integrator hosts one connector
                       https://connector.example.com
                                    ^
          credentials A on          |          credentials B on
          every request             |          every request
                                    |
   +------------------------+       |       +------------------------+
   | Northwind Trading      |-------+-------| Harbor Logistics       |
   | their Atlas, their     |               | their Atlas, their     |
   | source, their secrets  |               | source, their secrets  |
   +------------------------+               +------------------------+
             |                                          |
             v                                          v
   accounts, contacts, deals              vessels, ports, voyages, incidents

Two things make this work. The connector's public document declares a credentialSchema, the exact inputs Atlas asks the person connecting to fill in. Then every authenticated request body carries a credentials object with those values, and that object is the only place your code reads them from.

Never read upstream credentials from the environment

The only secret in a connector's environment is the bearer token Atlas presents. Database URLs, API keys, and app secrets belong to the tenant and arrive as req.credentials on each call. A connector that reads them from process.env serves exactly one tenant and cannot be connected twice.

Step zero: credentialSchema and check()

The first thing an author writes is the answer to two questions: what does a tenant have to type to reach their own instance, and what is the cheapest upstream call that proves they typed it right.

The Lark Base connector in the SDK's examples/lark asks for an app id, an app secret, and the token of the base to read. type: "password" masks the input in the connect form, type: "textarea" gives it the multi-line box a pasted key needs, and required: false marks a field a tenant may leave blank:

src/capability.ts
  // per tenant: the app credentials and the base they open
  credentialSchema: [
    {
      key: "appId",
      label: "App ID",
      type: "text",
      required: true,
      placeholder: "cli_XXXXXXXXXXXXXXXX",
      help: "Lark Developer Console → your app → **Credentials & Basic Info**, the field labelled **App ID**. Your apps are listed at [open.larksuite.com/app](https://open.larksuite.com/app).",
    },
    {
      key: "appSecret",
      label: "App secret",
      type: "password",
      required: true,
      help: "The **App Secret** on that same **Credentials & Basic Info** page of the [Lark Developer Console](https://open.larksuite.com/app). The app also needs the `bitable:app:readonly` permission.",
    },
    {
      key: "appToken",
      label: "Base app token",
      type: "text",
      required: true,
      placeholder: "bascnXXXXXXXXXXXXXXXXXXXXXX",
      help: "The id in the base's URL, `https://<tenant>.larksuite.com/base/<app_token>`. Add the app to that base as a collaborator first, or it cannot read the tables.",
    },
  ],

help is short markdown, and it is the part a tenant actually reads: name the exact page in the vendor's console the value is copied from and link the vendor's own doc for it, rather than restating the label. The connect form renders it above the input, with placeholder shown inside the empty field. A field left out of required altogether is required: the flag only ever loosens. An optional field a tenant leaves blank arrives with its key absent from credentials, so read it as missing rather than as an empty string.

check(req) receives those values on req.credentials, plus the host's own req.timeoutMs, and throws if they are wrong. Its message is the one place a connector speaks to the tenant directly: Atlas shows it verbatim next to Test connection, so write it for the person who typed the credentials.

src/connector.ts
  async check(req: CheckRequest): Promise<void> {
    await clientFor(req.credentials).checkAccess(makeDeadline(req.timeoutMs));
  }

Lark's checkAccess mints a tenant token (which proves the app id and secret) and reads the base's table list with page_size: "1" (which proves the app token), so either half being wrong fails here with Lark's own message.

For a SQL source SqlConnector supplies both halves: the default credentialSchema is one masked databaseUrl input, and check runs SELECT 1 on the pool opened from it.

The class, by plane

You extend AtlasConnector. Its methods fall into four planes, and the class file is laid out in that order. Every method receives the parsed wire request, including credentials and timeoutMs; serve() has already checked the bearer, validated the body, and started the deadline.

Identity

memberwireAtlas uses it forcostdefaultoverride when
slugin the documentnaming the connector; must match the document's slugnoneabstractalways: you declare it
capability()GET /.well-known/futurity/atlas.jsonwhich pushdowns it may use and which credentials to ask for; read on connect and refreshnoneabstract (derived for SQL)always on the REST path
check(req)POST /checkTest connection, and every conformance runone round tripabstract (SELECT 1 SQL)always on the REST path

The document is fetched unauthenticated and read on every connect and rediscovery, so build it from constants.

Query

methodwireAtlas uses it forcostdefaultoverride when
query(req)POST /queryevery row Atlas reads: previews, exports, local joins, host-side countsone scan under the request's filterabstractalways
count(req)POST /countpagination and preview sizingone counting callabsentthe source counts under a filter
aggregate(req)POST /aggregategroup-by pushdown Atlas tries before folding rows itselfone grouped callabsentthe source groups server-side

query is the only route that moves rows, and it is always a stream. It returns AsyncIterable<SourceRow[]>; each batch frames as an NDJSON line, with the heartbeats and the terminator around it. Yield batches of at most 5,000 rows; a larger batch is re-chunked on the wire.

count and aggregate are mounted only when you override them, and only what the document's endpoints lists is ever posted. Anything you leave out, Atlas answers itself over query inside its own reach, so a connector that pushes nothing is still correct, only slower.

Two laws hold on this plane.

The filter law. Every row you answer must satisfy every filter you declared an operator for. A filter naming a field that does not exist is refused with a 422, never answered with unfiltered rows. A filter you simply do not push is different: answer the superset and Atlas judges the rest itself, because operators is what it plans against. Which of the two you did is what the stream's first line says, served.filters true only when every predicate held. The kit's assertKnownFields(req, fieldNames) throws the 422 for an unknown field; call it in query and count before any fetch. SqlConnector refuses an unknown column on its own.

The aggregate rule. A wrong number is never legal. aggregate returns rows or undefined, and undefined becomes a 204 that tells Atlas to fold the rows itself. Add "aggregate" to the document's endpoints only when you override the method; serve() warns at boot when the two disagree.

Discovery

methodwireAtlas uses it forcostdefaultoverride when
discover(req)POST /discoverythe tables, fields, keys, and foreign keys a source is built from; called at setup and on rediscoverywhatever metadata costs; may be slowabstractalways: only you know the shape upstream

The answer is { tables, warnings? }. Each table names its primaryKey, its foreignKeys, and its fields with an Atlas type, nullable, unique, a few samples, and a sourceDescription. Descriptions travel into Atlas's understanding of the source, a declared key that is not really unique poisons joins, and a foreign key you forget is a join Atlas never plans, so this is the method that deserves the most care.

Size and measurement

Atlas needs one number before it plans anything: how big an entity is. It asks size once, in one request, and compares the answer against what it reaches in a single query. Past that it refuses, naming the rows and the routes, rather than half-answering.

The other two measurements are lazy. Atlas takes them when a question needs a fact your discovery did not declare, and it takes them itself over query when you do not serve the route.

methodwireAtlas uses it foroverride when
size(req)POST /sizethe reach check before any pull, and the row count a tenant seesthe source knows its size; exact: false when it only estimates
cardinality(req)POST /cardinalitynon-null and distinct counts per column; key promotion picks join keys from themthe source has COUNT DISTINCT; answer null per column it cannot count
linkHitRate(req)POST /linkHitRatethe orphan rate of a candidate foreign key; keeps or drops the join hopthe source can LEFT JOIN

An entity's grain is derived, never asked for: one row per business object means a column whose distinct count equals its non-null count equals size.

Every route here is mounted only when you override it, and Atlas posts only what the document's endpoints lists. Leave size out and Atlas sizes the entity with a bounded pull of its own; leave the other two out and it measures them the same way. cardinality is judged against the row count size carries, so a connector that serves cardinality without size gets sized by that pull and keeps whatever columns it did answer, the pull filling in only the ones it left out. Conformance has no row count to judge the grain against there and skips it, so serve the pair.

Two paths: SQL and REST

A SQL database: extend SqlConnector

When the data lives in a SQL engine the SDK writes the SQL. You declare a catalog and provide three things: how to open a pool from a tenant's credentials, how to close one, and how to run one parameterized statement. The reference connector Brightline, a CRM over Postgres in examples/brightline-crm, is exactly that:

src/connector.ts
  protected override async openPool(credentials: Credentials): Promise<SQL> {
    if (!credentials.databaseUrl) throw new Error("databaseUrl is required");
    return new SQL(pinnedUrl(credentials.databaseUrl));
  }

  protected override async closePool(pool: SQL): Promise<void> {
    await pool.close();
  }

  // the only path SQL text reaches pg on; values never travel inside the statement string
  async run(pool: SQL, sql: string, params: unknown[]): Promise<Row[]> {
    return (await pool.unsafe(sql, params)) as Row[];
  }

SqlConnector keeps one pool per credential set, keyed by a hash of the credentials so key order never matters, and holds at most 16; the least recently used pool is closed off the request path when a seventeenth tenant arrives, and a pool a request is still reading from stays open until that request returns. From catalog and run it derives every protocol method: check, discovery, the query stream, counts, sizes, column cardinality, link hit rates, and GROUP BY pushdown. The capability document is computed from the catalog and the flavor, so it cannot advertise an operator the builders cannot render, and credentialSchema defaults to the single databaseUrl input.

The catalog is a typed literal. Each col(name, wire, type, opts?) names the column, how its storage crosses the wire (int, decimal, text, boolean, date, datetime, text_array), and the Atlas type discovery reports:

src/catalog.ts
  {
    name: "deals",
    description: "Opportunities. company_id/owner_id are enforced pg FKs.",
    primaryKey: ["id"],
    foreignKeys: [
      { field: "company_id", targetTable: "companies", targetField: "id" },
      {
        field: "primary_contact_id",
        targetTable: "contacts",
        targetField: "id",
      },
      { field: "owner_id", targetTable: "owners", targetField: "id" },
    ],
    columns: [
      col("id", "int", "number"),
      col("company_id", "int", "number"),
      col("primary_contact_id", "int", "number", { nullable: true }),
      col("owner_id", "int", "number"),
      col("name", "text", "string"),
      col("stage", "text", "string"),
      col("amount", "decimal", "decimal", { nullable: true }),
      col("currency", "text", "string"),
      col("expected_close", "date", "date", { nullable: true }),
      col("closed_at", "datetime", "datetime", { nullable: true }),
      col("created_at", "datetime", "datetime"),
    ],
  },

Set unique: true only for a real database UNIQUE or primary-key constraint, and set keysEnforced = true on the class only when every declared key is one. Three seams exist for when the defaults run out: flavor (Postgres is the shipped dialect), credentialSchema (override when the driver takes separate parts instead of one URL), and streamBatches, which lets a driver with real cursors replace the limit/offset paging query otherwise uses. Brightline overrides streamBatches with a DECLARE ... CURSOR and FETCH FORWARD 5000.

A REST or ERP API: extend AtlasConnector

When the source is an API there is no SQL to derive from, so you extend AtlasConnector directly and write three methods: check, query, and discover, plus size wherever the API knows how big a table is. Everything else is optional, and what you leave out Atlas answers itself. The document is authored by hand, and every flag in it is earned: an advertised operator your query silently drops corrupts answers downstream.

The work concentrates in one decision: what pushes down to the API and what runs in memory. Most business APIs filter on a few fields and no more. Push those, then run the full filter set through applyFilters, the in-memory twin of the SDK's SQL where-builder. Both routes count as honoring an operator. Lark's scan does this, and refuses the field it cannot evaluate first:

src/connector.ts
  // pushes the pushable slice of and[]; rows carry exactly the needed columns
  private async *scan(client: LarkClient, req: QueryShape, deadline: Deadline): AsyncIterable<SourceRow[]> {
    if (req.joins && req.joins.length > 0) throw unsupported("joins are not supported; atlas joins locally");
    const { meta, table } = await this.resolveTable(client, req.table, deadline);
    const fieldsByName = await this.fields(client, meta, table.table_id, deadline);
    assertKnownFields(req, [RECORD_ID, ...fieldsByName.keys()]);
    const columns = neededColumns(req);
    const realFields = [...columns].filter((column) => fieldsByName.has(column));
    const batches = client.searchAll(table.table_id, deadline, {
      // field_names must name real fields; record_id rides along on every record anyway
      fieldNames: realFields.length > 0 ? realFields : undefined,
      conditions: pushdownConditions(req.and, fieldsByName),
    });
    for await (const records of batches) {
      yield records.map((record) => toRow(record, columns, fieldsByName));
    }
  }

  // scan batches with the FULL filter set re-applied locally; batches may come out empty
  private async *scanFiltered(client: LarkClient, req: QueryShape, deadline: Deadline): AsyncIterable<SourceRow[]> {
    for await (const batch of this.scan(client, req, deadline)) {
      yield applyFilters(batch, { and: req.and, or: req.or }, req.fieldTypes);
    }
  }

Lark's document advertises only the operators the API really pushes, and Atlas judges the rest over the rows it answers. It declares join: false so Atlas joins hops locally, and endpoints: ["size"] because every search page already carries the table's total while a base has no server-side group-by, no distinct count, and no join to measure a link with.

The rows you return are SourceRow values, Record<string, string | number | boolean | null>, and the wire spellings are exact: decimals and integers past 2^53 cross as digit-exact strings, datetimes as ISO-8601 UTC, arrays and JSON as JSON text inside a string, missing as null. Atlas validates every answer and refuses a malformed one rather than coercing it.

From scaffold to connected source

Scaffold it

bun create atlas-connector stamps a working project for either path:

$ bun create atlas-connector erp-bridge --kind rest --port 4200
Scaffolded 'erp-bridge' (rest) at /home/you/erp-bridge

Next steps:
  cd erp-bridge
  cp .env.example .env    # set ATLAS_CONNECTOR_TOKEN to a 32+ char secret
  bun install
  bun run start           # serves on :4200

Then fill in the YOUR CODE HERE methods in src/connector.ts; earn each flag in src/capability.ts, and point atlas-conform at it to grade the result.

Run it bare and it prompts for the name, the kind (a sql database or a rest or erp api), and the port. The sql template is a working Postgres connector as stamped: openPool, closePool, and run over Bun's SQL, and the only thing to fill in is src/catalog.ts. The rest template stamps check, query, count, and discover as YOUR CODE HERE stubs, each with its one-line contract, calls assertKnownFields in query and count already, and ships a deliberately narrow src/capability.ts. Both pin the SDK at the version the CLI shipped with.

src/index.ts reads ATLAS_CONNECTOR_TOKEN and the port and calls serve(new MyConnector(), { token, port }), which refuses to boot on a token under 32 characters or a capability document that does not parse.

Run it locally

Start it and read the document, then prove a credential set against /check with the same body shape Atlas will send:

curl http://localhost:4100/.well-known/futurity/atlas.json

curl -X POST http://localhost:4100/check \
  -H "authorization: Bearer $ATLAS_CONNECTOR_TOKEN" -H 'content-type: application/json' \
  -d '{"credentials":{"databaseUrl":"postgres://user:pass@localhost:5432/mydb"},"timeoutMs":5000}'

A good set answers {"ok":true}. A bad one answers HTTP 400 with {"error":{"code":"check_failed","message":"..."}}, where the message is whatever your check threw.

Prove it with atlas-conform

atlas-conform is the conformance program packaged as a CLI, so you can grade a connector before anyone connects it. Atlas runs the same program in live mode after every connect, so a verdict here is the verdict that lands on the source. The runner ships from Futurity rather than a public registry: ask your Futurity contact for it, or connect the source in a test organization and read the report Atlas records, which carries the same check ids.

atlas-conform --url https://connector.example.com --token "$ATLAS_CONNECTOR_TOKEN" \
  --mode live --credentials '{"databaseUrl":"postgres://..."}'

It exits 0 on pass, 1 when any check fails, and 2 when it could not run at all (the document would not fetch, or the fixture corpus is not loaded).

--credentials takes inline JSON or @path/to/file.json; a connector whose document declares an empty credentialSchema needs neither. --token can be replaced by the ATLAS_CONNECTOR_TOKEN environment variable, and --report out.json writes the full report.

There are three modes. fixture (the default) runs phases 0 to 3 and compares your answers against a DuckDB oracle over a known corpus you load into your source first with atlas-conform load --url <dsn>. live runs phases 0, 1, 3 and 4 read-only against your real data, and is what Atlas runs. both runs everything. --phases 0,1,3 narrows a run, and --row-budget (default 200,000) keeps the live streaming checks off tables larger than that.

phaseid prefixwhat it proves
0P0.xthe document is served unauthenticated, parses, does not redirect; every route rejects an empty body with a 400 envelope
1P1.xno token, a wrong token, and a query-string token all 401 without leaking; the right one 200s; /check answers { ok: true }
2P2.xfixture agreement: discovery types, every operator, sort, paging, counts, streams, aggregates, measurement-to-join agreement, wire type fidelity
3P3.xbehaviour: deadlines honored, unknown entity 404, malformed body 400, unknown filter field 400 or 422, stream bounds and heartbeats, joins refused rather than silently dropped
4L1 to L10live self-consistency: an exact size equals the streamed rows, filters narrow, measured counts stay ordered, declared keys are unique, pages compose

Each line of the report is one verdict: a pass, a fail with the detail and the request that produced it, or a skip. Skips are not failures: a check skips when your document does not declare the thing it tests (sort: "none", no aggregate endpoint, join: false) or when the data cannot exercise it.

Deploy it

Any host that runs Bun and gives you a public HTTPS address works. Atlas calls the connector from its own servers and refuses hosts that resolve onto a private network, so localhost is only for atlas-conform. The connector's environment holds exactly two things: the port (the examples read PORT, then CONNECTOR_PORT, then default to 4100) and ATLAS_CONNECTOR_TOKEN. Nothing about any tenant's upstream goes there.

Redeploy after changing secrets

Changing a variable in a host's dashboard does not always reach a running container. After you set or rotate ATLAS_CONNECTOR_TOKEN, force a fresh deploy, then re-read the well-known document and re-run /check before trusting it.

Connect it in Atlas

In Select Data Sources, the External Connector card appears when you hold the Manage Atlas permission (the Atlas External Connectors feature is on unless an administrator switched it off for your organization). The form takes a name, the connector's base URL, and the bearer token you issued. The base URL is any https prefix, path included, so a deployment hosting several connectors gives each one its own path.

The Token field is the connector's own bearer token, the value its host set as ATLAS_CONNECTOR_TOKEN. It proves to the connector that the caller is Atlas, so it is issued by whoever hosts the deployment and is shared by every client connecting to that deployment. It is not one of the credentials that open your own upstream data; those are the fields the form renders once the document is read. If an integrator hosts the connector for you, they are who you get this token from. If you host it yourself, you mint it (openssl rand -hex 24 gives 48 characters) and set it as ATLAS_CONNECTOR_TOKEN; nothing is issued by Futurity.

Fill in the base URL and the token, then click Read connector. Atlas fetches the public document and checks the token against your /check; a bearer the connector refuses fails the read with "The connector rejected the token". On success it shows the slug it found, and renders one input per credentialSchema entry: masking the password ones, giving a textarea one a multi-line box, marking anything the connector declared optional, and showing each field's help above it. Changing the URL forgets the document and the values typed against it. Changing the token forgets the document, since the read is what judges the token, and keeps what you typed.

The External Connector form after Read connector: slug lark-base confirmed, with the App ID, App secret and Base app token inputs rendered below the token, each carrying its own placeholder and help line

Test connection is enabled once the document is read, the token is 32 characters or more, and every required credential input has a value. It posts to your /check with those credentials. A rejected set shows your check message verbatim; a rejected bearer shows "The connector rejected the token".

Test connection in its failed state, showing the message lark tenant_access_token failed: code=10014 app secret invalid

A wrong App secret above reaches the form as lark tenant_access_token failed: code=10014 app secret invalid, which is Lark's own wording carried through check() untouched. Write your message the same way: name what was rejected and what the upstream said about it.

Test connection in its accepted state: Credentials accepted, with Add source enabled

Click Add source. Atlas caches the document, marks the source verifying, and a background worker runs atlas-conform in live mode with the credentials you typed. The verdict lands on the source's status chip as conformant or failed conformance; a failed run keeps the first ten failures on the source, each with its check id, and the chip's tooltip carries the detail of the first. The circular-arrows button on a connected connector row, labelled Rebind, reopens the same form with a Reconnect button: it re-reads the document, takes a fresh credential set, and runs conformance again, which is how you pick up a connector that changed the entities or fields it exposes.

From there the connector is discovered and queried like any other kind. Discovery asks each table for its size and reads one page of rows; anything Atlas has to measure itself is a WAN round trip to your service, so it takes those lazily, when a question needs the fact.

The same connector, twice

Multitenancy is not a claim the document makes; it is observable. The SDK's examples/lark/seed fills two Lark bases with two unrelated datasets: tenant A, Northwind Trading, a trading company with accounts (30 rows), contacts (80) and deals (60); and tenant B, Harbor Logistics, a shipping company with vessels (12), ports (15), voyages (90) and incidents (20). The two share no table names on purpose, so a leak between tenants would be visible at a glance.

One deployed connector, with no Lark credentials in its environment, is connected into Atlas twice: once with Northwind's app credentials and base token, once with Harbor's. Both pass Test connection, and swapping Northwind's secret for a wrong one fails it.

Select Data Sources listing Harbor Logistics and Northwind Trading, both External Connector, both with a green conformant chip

Discovery for Northwind lists accounts, contacts, and deals and nothing else. Discovery for Harbor lists vessels, ports, voyages, and incidents. A query on each returns its own row counts, and asking the Northwind source for a deals table works while asking Harbor for one answers 404 unknown_entity. The process serving both holds a metadata cache keyed by the whole credential set, so a tenant who cannot open a base never reads its tables out of another tenant's entry.

Atlas names an entity per discovered table, so the two catalogs stay separate all the way into the entity list:

The Atlas entity list grouped by source: Harbor Logistics holding Incident, Port, Vessel and Voyage; Northwind Trading holding Account, Contact and Deal

Ask Corint how many Northwind deals are won and it plans over the Deal entity, counts through the connector, and answers 11, the number seeded into that base and no other:

A Corint chat: the question about won deals, three Atlas tool steps, and the answer 11

Debugging and common pitfalls

Most connector problems surface the same way: atlas-conform fails a check, or a live query returns the wrong rows. These are the ones you are most likely to hit.

Reject filters on fields you do not have

The single most common bug on the AtlasConnector path. applyFilters treats an unknown filter field as a missing value: it silently matches nothing instead of erroring, and the empty answer looks like a correct one. The protocol requires a filter on a field the table does not have to be refused. SqlConnector does this for you; the hand-written path must call the kit:

import { assertKnownFields } from "@futurity/atlas-connector";

  async *query(req: NativeQueryRequest): AsyncIterable<SourceRow[]> {
    // a filter you cannot answer must 422 HERE: a row that skipped a filter reads as a row that matched it
    assertKnownFields(req, fieldsOf(req.table));

This passes every hand test

Your own filters always name real fields, so the bug only shows up as conformance check P3.4c, "an unknown filter field → 400/422", reporting got no-error. Add the guard to query and count before you write your first fetch.

Read a conformance failure

A failed run in Atlas stores the first ten failures on the source; a CLI run prints every verdict and can write the whole report with --report. Each failure names the check id, its title, and what it saw:

{
  "mode": "live",
  "summary": { "pass": 31, "fail": 1, "skip": 2 },
  "failures": [
    {
      "id": "P3.4c",
      "title": "an unknown filter field → 400/422",
      "detail": "got no-error"
    }
  ]
}

Each id maps to one protocol obligation from the phase table above, and detail states the gap in plain terms. Fix that one check and re-run; the CLI report also carries the exact request body that produced a failure.

When a live call fails and Atlas shows only internal

The wire is sanitized on purpose. Anything your method throws that is not a ConnectorError reaches Atlas as {"error":{"code":"internal","message":"internal error"}}, never your stack. serve() logs the real error to the connector's own stderr first, prefixed with the slug, so read your host's logs for the cause. Most first failures are the upstream rejecting a credential, a wrong base URL, or a response shape that differs from what you parsed. Hit the upstream API directly with curl and compare its real response to what your code assumes.

To answer with a specific status instead, throw one of the SDK's constructors: badRequest, unauthorized, unknownEntity, unsupported, or timeout. Lark, for example, maps its API's not-found code to unknownEntity, which is what makes an unknown table a 404 rather than a 500.

Wire reference

Every route is relative to the base URL a source is registered with, which is any https prefix. A connector mounted at https://atlas.example.com/lark-base serves its document at https://atlas.example.com/lark-base/.well-known/futurity/atlas.json and its routes under the same prefix, so one host can carry several connectors.

Every route except the well-known document is a POST with a JSON body and the header Authorization: Bearer <token>. Every authenticated body carries credentials, a flat object of strings, and timeoutMs, a positive integer the server enforces with a 408. Every JSON answer is a wrapped object, never a bare array. Values in rows are string | number | boolean | null.

GET /.well-known/futurity/atlas.json

Unauthenticated. At most 64 KiB, content-type: application/json, no redirect. Unknown top-level fields are stripped by the reader.

{
  "protocolVersion": 1,
  "slug": "lark-base",
  "capabilities": {
    "operators": [
      "eq",
      "neq",
      "gt",
      "gte",
      "lt",
      "lte",
      "in",
      "nin",
      "contains",
      "includes",
      "startswith",
      "isnull",
      "notnull"
    ],
    "dateBucket": false,
    "sort": "multi",
    "offset": true,
    "join": false,
    "keysEnforced": true,
    "limits": { "pageSizeMax": 500, "rowsPerTableMax": 20000, "concurrency": 1 }
  },
  "credentialSchema": [
    {
      "key": "appId",
      "label": "App ID",
      "type": "text",
      "required": true,
      "placeholder": "cli_XXXXXXXXXXXXXXXX",
      "help": "Lark Developer Console → your app → **Credentials & Basic Info**, the field labelled **App ID**. Your apps are listed at [open.larksuite.com/app](https://open.larksuite.com/app)."
    },
    {
      "key": "appSecret",
      "label": "App secret",
      "type": "password",
      "required": true,
      "help": "The **App Secret** on that same **Credentials & Basic Info** page of the [Lark Developer Console](https://open.larksuite.com/app). The app also needs the `bitable:app:readonly` permission."
    },
    {
      "key": "appToken",
      "label": "Base app token",
      "type": "text",
      "required": true,
      "placeholder": "bascnXXXXXXXXXXXXXXXXXXXXXX",
      "help": "The id in the base's URL, `https://<tenant>.larksuite.com/base/<app_token>`. Add the app to that base as a collaborator first, or it cannot read the tables."
    }
  ],
  "endpoints": []
}
fieldtypenotes
protocolVersionliteral 1any other value fails the parse
slugstring^[a-z][a-z0-9-]{2,39}$; must not collide with a built-in source kind
dialectstring, optionalpresent only for a dialect-mode connector that accepts compiled SQL; absent for native mode
capabilities.operatorsarray of Op, possibly emptythe filter operators query and count push upstream; [] means Atlas judges every filter itself
capabilities.dateBucketbooleanwhether aggregate accepts a grain on a group-by
capabilities.sort"none", "single", "multi"how many sort keys query honors
capabilities.offsetbooleanwhether query honors offset
capabilities.joinbooleanwhether query executes joins; false means Atlas joins hops locally
capabilities.keysEnforcedbooleanevery declared primary key and unique field is a real source constraint
capabilities.limitsobjectthe vendor's own ceilings: pageSizeMax, rowsPerTableMax, concurrency (1 to 16), offsetMax; Atlas plans its pulls against them
credentialSchemaarray of credential fieldseach is { key, label, type, required } plus optional placeholder and help; empty only when the source needs no per-tenant secret
credentialSchema[].type"text", "password", "textarea"password masks the input; textarea is the multi-line box a pasted key needs
credentialSchema[].requiredboolean, defaults to truea required field gates Test connection and submit; a blank optional one is absent from credentials
credentialSchema[].placeholderstring, optionalexample value shown inside the empty input
credentialSchema[].helpstring, optionalshort markdown above the input, naming where the value is found
endpointsarray of optional route names"size", "count", "aggregate", "cardinality", "linkHitRate"; the SDK fills it from the methods you override

Filters, sorts, and joins

A Filter is one of three shapes, each strict:

{ "field": "stage", "op": "eq", "value": "won" }
{ "field": "stage", "op": "in", "values": ["won", "lost"] }
{ "field": "closed_at", "op": "isnull" }

Value ops are eq, neq, gt, gte, lt, lte, contains, includes, startswith and carry a scalar value. Member ops are in and nin and carry an array values. Nullary ops are isnull and notnull. and is a conjunction; or is an array of groups in disjunctive normal form, and the whole or block is one more conjunct.

A sort is { "field": "amount", "dir": "asc" } with an optional "collate": true on a compiled sort. A join hop is { "fromTable", "toTable", "fromField", "toField", "fields": [{ "field", "as", "type" }] }. fieldTypes is an optional record of field name to Atlas type (string, number, decimal, boolean, date, datetime, json, array, reference) and is how applyFilters knows to compare digits rather than bytes.

POST /check

Request: { "credentials": {...}, "timeoutMs": 5000 }.

Answer: { "ok": true }. A rejected credential set answers HTTP 400 with { "error": { "code": "check_failed", "message": "<what check threw>" } }.

POST /discovery

Request: { "credentials": {...}, "timeoutMs": 15000 }.

Answer: { "tables": [...], "warnings": ["..."] } where warnings is optional and each table is:

{
  "name": "deals",
  "sourceDescription": "lark base table",
  "rowCount": 60,
  "storesRows": true,
  "primaryKey": ["record_id"],
  "foreignKeys": [
    {
      "field": "account",
      "targetTable": "accounts",
      "targetField": "record_id"
    }
  ],
  "fields": [
    {
      "name": "record_id",
      "sourceColumn": "record_id",
      "type": "string",
      "nullable": false,
      "unique": true,
      "samples": ["recuA1", "recuA2"],
      "sourceDescription": "lark record id (system primary key)",
      "stats": {
        "nullPercent": 0,
        "distinctCount": 60,
        "min": "recuA1",
        "max": "recuZ9"
      },
      "filterable": true,
      "groupable": true,
      "aggregatable": false
    }
  ]
}

rowCount, stats (with optional nullPercent, min, max and a required distinctCount), filterable, groupable, and aggregatable are optional; everything else is required.

POST /query

Request, native mode:

{
  "table": "deals",
  "and": [{ "field": "stage", "op": "eq", "value": "won" }],
  "sort": [{ "field": "amount", "dir": "desc" }],
  "limit": 100,
  "offset": 0,
  "fields": ["record_id", "name", "amount"],
  "fieldTypes": { "amount": "number" },
  "credentials": {},
  "timeoutMs": 30000
}

table, and, sort, fields, credentials, and timeoutMs are required; or, limit, offset, joins, and fieldTypes are optional, and the body also carries idleTimeoutMs and maxTimeoutMs. Atlas sends an idle timeout of 30 seconds, three missed heartbeats. In dialect mode the query fields are replaced by { "sql": "...", "params": [...] }.

Answer: content-type: application/x-ndjson, one JSON object per line:

{ "served": { "filters": true, "sort": true, "window": false } }
{ "rows": [ { "record_id": "recuA1" } ] }
{ "ping": 1 }
{ "end": 1 }
{ "error": { "code": "timeout", "message": "no rows within idle deadline" } }

served is the first line, written before any rows, and it names which parts of the request this answer already applied. filters means every and and or predicate held, with Atlas's semantics. sort means the rows arrive in the order the request asked for. window means offset and limit were applied, with no row beyond them on the stream. A stream that omits the line served none of the three, which is always safe: Atlas finishes what is left over the rows it receives. window is read only when filters and sort are served with it, because a page cut from the wrong rows in the wrong order is the wrong page; ndjsonStream clamps it for you, and counts a request that asked for no order as sorted. This line is where the filter law is answered: push every predicate and say filters: true, or answer the superset and say filters: false.

A rows line carries between 1 and 5,000 rows. ping is written after 10 seconds without a batch. end is written once, after the producer completes. error is terminal with no end. A stream that closes with neither end nor error is truncated and the reader fails it.

POST /count

Request: { "table", "and", "or"?, "fieldTypes"?, "credentials", "timeoutMs" }, the filter subset of /query. Mounted only when count() is overridden.

Answer: { "count": 11 }.

POST /aggregate

Request:

{
  "table": "deals",
  "and": [],
  "or": [],
  "groupBy": [{ "field": "closed_at", "as": "month", "grain": "month" }],
  "measures": [
    { "fn": "sum", "field": "amount", "as": "total" },
    { "fn": "count", "as": "n" }
  ],
  "stringFields": ["stage"],
  "joins": [],
  "fieldTypes": { "amount": "decimal" },
  "limit": 1001,
  "credentials": {},
  "timeoutMs": 30000
}

fn is one of count, sum, min, max, count_distinct; avg is never pushed. grain is one of year, quarter, month, day and is sent only to a dateBucket: true connector. limit is the group budget plus one: Atlas sends 1,001 so that a 1,001st group proves overflow, and it discards an answer that long rather than reporting a truncated set, so refuse rather than truncate. The number never varies with the question: a top-N is cut from the groups Atlas gets back, never asked for on this route.

Answer: { "rows": [ { "month": "2026-03-01", "total": "48210.50", "n": 7 } ] }, or HTTP 204 with no body to decline this aggregate.

POST /size

Request: { "table": "deals", "credentials": {}, "timeoutMs": 15000 }.

Answer: { "size": { "rows": 60, "exact": true } }, exact: false when the source only estimates, or { "size": null } when it cannot say at all. The null sits inside the wrapper, never as a bare body.

POST /cardinality

Request: { "table": "contacts", "columns": ["email", "company_id"], "credentials": {}, "timeoutMs": 15000 }, with at least one column.

Answer:

{
  "columns": {
    "email": { "nonNull": 80, "distinct": 79 },
    "company_id": null
  }
}

A column you cannot count distinctly answers null, and Atlas measures that one itself over query. The counts are judged against size, so a source that serves this route should serve that one too.

POST /linkHitRate

Request: { "fromTable": "contacts", "fromColumn": "company_id", "toTable": "accounts", "toColumn": "record_id", "credentials": {}, "timeoutMs": 15000 }.

Answer: { "fromNonNull": 78, "orphanCount": 1, "orphanRate": 0.01282, "orphanSamples": ["recuMissing"] }, with at most 20 samples.

The error envelope

Every non-2xx JSON body is exactly { "error": { "code": "...", "message": "..." } }.

statuscodewhen
400bad_requestthe body did not parse against the route's schema
400check_failed/check only: the credentials were rejected; the message is the author's
401unauthorizeda missing or wrong bearer
404unknown_entitya table or column the source does not have
408timeoutthe request's own timeoutMs elapsed
422unsupporteda legal Atlas request the document never advertised, or a filter field the connector cannot evaluate
500internalanything else; the message is always internal error and the real error is logged server-side

Limits

limitvalue
capability document64 KiB
JSON answer body32 MiB
one NDJSON line16 MiB
rows per stream batch5,000
heartbeat interval10 s
Atlas's stream idle timeout30 s
bearer token32+ chars

Where to go next