Introduction

External data connectors allow Phonemos to integrate structured data from external systems like Jira, YouTrack, GitHub, and other issue trackers or data sources. Connectors provide a standardized way to synchronize entities (issues, projects, users, etc.) from external systems into Phonemos, enabling users to search, link, and work with external data alongside Phonemos content.

A connector is a server that runs independently and implements the Phonemos connector API. The Phonemos backend communicates with connectors via REST API calls to fetch data change events and manage data sources.

Architecture Overview

Core Concepts

Connector: A server that implements the connector API. Connectors are typically instantiated per connected system type (e.g., one connector for Jira, one for YouTrack). A connector makes one or more data sources available.

Data Source: Represents a connection to a specific external system instance (e.g., a particular Jira server at jira.mycompany.com). Each data source is configured with connection details and credentials. A connector can manage multiple data sources simultaneously.

Entity: An item within the external system (e.g., an issue, a project, a git commit, a user). Entities have:

• A unique identifier (type, instance, and local ID)

• Fields (attributes like title, status, priority)

• References (relationships to other entities)

Event: A change to an entity. Events can be:

• Upsert: Insert or update an entity with all its current field values and references

• Delete: Mark an entity as deleted (tombstone)

Cursor Position: A position marker that tracks progress through the event stream. Phonemos uses cursor positions to resume fetching events after the last processed position.

Data Flow

1. Phonemos creates a data source by calling PUT /v1/connector/data-sources/{id} with configuration

2. Phonemos queries the data source schema via GET /v1/connector/data-sources/{id}/info

3. Phonemos periodically polls GET /v1/connector/data-sources/{id}/events with an afterPosition parameter

4. The connector returns events starting after the given position

5. Phonemos processes events and updates its internal data model

6. When a data source is deleted, Phonemos calls DELETE /v1/connector/data-sources/{id}

API Specification

The connector API is defined by the OpenAPI 3.1.0 specification: External Data Connector Swagger Specification. All connectors must implement the following endpoints:

Authentication

All endpoints require authentication via the X-Api-Key header. The API key is configured when Phonemos connects to your connector. Your connector should validate this header and return 403 Forbidden if the key is missing or invalid.

Endpoints

GET /v1/connector/info

Returns metadata about the connector and its capabilities.

Response: ConnectorMetadata

• kind: String identifier for the connector type (e.g., "jira", "youtrack")

• label: Human-readable name for the connector

• version: Version string of the connector

• configSchema: Schema defining what configuration options are available for data sources

Example Response:

PUT /v1/connector/data-sources/{id}

Creates or reconfigures a data source. Phonemos calls this when a user creates or updates a data source configuration.

Path Parameters:

• id: Unique identifier for the data source (chosen by Phonemos, typically a UUID)

Request Body:

The config object structure is defined by your connector's configSchema. You should validate the configuration and return 400 Bad Request with error details if invalid.

Responses:

• 200: Data source created or reconfigured successfully

• 400: Invalid configuration (include error details in response body)

• 500: Internal connector error

Error Response Format:

DELETE /v1/connector/data-sources/{id}

Deletes a data source. Phonemos calls this when a user removes a data source. The connector should clean up any resources associated with the data source.

Responses:

• 200: Data source deleted successfully

• 404: Data source was not known to the connector (this is acceptable)

GET /v1/connector/data-sources/{id}/info

Returns metadata about the data source, including the schema of entities it provides.

Response: DataSourceInfo

• id: The data source ID (from path parameter)

• label: Human-readable label for the data source

• kind: Connector kind (same as connector info)

• initialPosition: The cursor position before the first event (the "zero" position)

• entities: Array of entity definitions

Example Response:

Responses:

• 200: Data source info returned

• 404: Data source not found (Phonemos will retry PUT to recreate it)

GET /v1/connector/data-sources/{id}/status

Returns the current status of the data source.

Response: DataSourceStatus

• status: One of "Ok", "Error", or "Unreachable"

• details: Optional details about the status

• lastPosition: Optional cursor position of the last available event

Example Response:

Responses:

• 200: Status returned

• 404: Data source not found

GET /v1/connector/data-sources/{id}/events

Returns data change events starting after the given cursor position.

Query Parameters:

• afterPosition (required): Cursor position to start fetching from (exclusive)

Response: Array of EntityEvent objects

Important Behaviors:

• Events must be ordered by position (ascending)

• The last event in the array should have the highest cursor position

• If no new events are available, you can either:

  • Return an empty array immediately (Phonemos will throttle the next request)

  • Wait up to 1 minute for new events to arrive, then return them

• Response size should not exceed 5MB

• You can return different numbers of events per request based on your logic

Example Response:

Responses:

• 200: Events returned (may be empty array)

• 404: Data source not found

REST API Implementation

Step 1: Set Up Your Server

Create a web server that listens on a configurable port. The server should:

• Accept HTTP requests

• Parse JSON request bodies

• Return JSON responses

• Handle CORS if needed

• Implement authentication via X-Api-Key header

Validate the X-Api-Key header on all requests and return 403 Forbidden if the key is missing or invalid.

Step 2: Implement Connector Info Endpoint

Return metadata about your connector including:

• kind: Unique identifier for your connector type

• label: Human-readable name

• version: Version string

• configSchema: Definition of configuration options with their names, descriptions, whether they're secret, and whether they're required

Step 3: Store Data Source Configurations

When Phonemos creates a data source via PUT /v1/connector/data-sources/{id}, store its configuration in memory or persistent storage. Validate the configuration against your config schema and return 400 Bad Request with error details if invalid.

Step 4: Implement Data Source Info Endpoint

Return the schema of entities your connector provides. This includes:

• Data source ID, label, kind, and initial position

• Entity definitions with their types, fields, and references

Each entity definition should include all fields with their types and all references with their target entity types.

Step 5: Implement Events Endpoint

This is the core of your connector. Fetch events from your external system starting after the given cursor position and return them as an array. Ensure events are ordered by position (ascending) and handle the case where no new events are available (return empty array or wait up to 1 minute).

Step 6: Implement Status Endpoint

Return the health status of your data source. Check connectivity to the external system and return status "Ok", "Error", or "Unreachable" with optional details and the last available cursor position.

Step 7: Implement Delete Endpoint

Clean up when a data source is deleted. Remove stored configurations and any other resources associated with the data source.

Data Model

Entity Identifiers

Each entity has a unique identifier consisting of three parts:

• type: The entity type (e.g., "issue", "user", "project")

• instance: Identifies the instance of the external system (e.g., "jira:1234-444-333-11" or "git:mycompany/my-repo")

• id: The local identifier within that instance (e.g., "PROJ-123" or a UUID)

Example:

The instance identifier should be stable and unique per external system instance. It's often a combination of the connector type and an instance ID.

Entity Definitions

When Phonemos queries /v1/connector/data-sources/{id}/info, you return entity definitions that describe the schema of entities your connector provides.

Fields

Fields define the attributes of an entity. Each field has:

• id: Unique identifier within the entity type (used in events)

• name: Human-readable name

• description: Description of what the field represents

• fieldType: The data type (see Field Types below)

References

References define relationships to other entities. Each reference has:

• id: Unique identifier within the entity type (used in events)

• name: Human-readable name

• description: Description of the relationship

• types: Set of entity types that can be referenced (e.g., ["user", "project"])

• multiple: Whether this is a one-to-many relationship

In events, references are provided as maps from reference ID to arrays of entity IDs (even for non-multiple references, the array just contains one element).

Field Types

Text

Plain text string. JSON value: string

Number

Numeric value. JSON value: number

Date

Date without time. JSON value: ISO-8601 date string (e.g., "2024-12-24")

Instant

Date and time with timezone. JSON value: ISO-8601 datetime string with timezone (e.g., "2023-04-04T15:20:30+02:00" or "2023-04-04T15:20:30Z")

Label

Categorical value(s). JSON value: array of strings (e.g., ["feature", "UI"]). Set multiple: true to allow multiple values, false for single value.

Link

A link to another resource. JSON value: object with label and uri keys

Example value:

Other

Any JSON value. Use when the field doesn't fit other types.

Events

Upsert Event

Inserts or updates an entity with all its current values. All fields and references must be provided. Fields not present will be set to null in Phonemos.

Delete Event

Marks an entity as deleted (tombstone).

Cursor Positions

Cursor positions track progress through the event stream. They must:

• Be serializable to strings

• Support ordering (so Phonemos can determine which events come after a position)

• Be stable (the same position should always refer to the same point in the stream)

Common cursor position strategies:

Sequential Numbers: Simple for systems with sequential IDs. Serialize as string representation of the number, deserialize by parsing the string back to a number.

Timestamps: For time-ordered events. Serialize as ISO-8601 datetime string, deserialize by parsing the ISO-8601 string.

Composite: Timestamp + ID for stable ordering when multiple events have the same timestamp. Serialize as JSON object with timestamp and optional ID fields, deserialize by parsing JSON.

Important: The initialPosition returned in data source info should be a position that comes before all events. For sequential numbers, use "0". For timestamps, use epoch or a very early date.

User Entities

The entity type "user" has special meaning in Phonemos. Users with type "user" are automatically synced to Phonemos user accounts. They must have these fields:

• email (required): Used to match users to Phonemos accounts

• first_name (optional): First name, only used if user doesn't exist in Phonemos

• last_name (optional): Last name, only used if user doesn't exist in Phonemos

• display_name (optional): Full display name if separate first/last names aren't available

Additional fields are allowed and will be stored.

If you want to return user-like entities that shouldn't be synced to Phonemos users, use a different entity type (e.g., "account", "contact").

Error Handling

Standard Error Format

All errors should follow this format:

Error Codes

Use these standard error codes:

• parameters: Invalid request parameters

• not-found: Resource not found

• unauthorized: Authentication failed

• invalid-config: Invalid data source configuration

• authentication-failed: Failed to authenticate with external system

• unreachable: External system is unreachable

• resource-not-found: Resource not found in external system

• external-system-error: Error returned by external system

• rate-limited: Rate limited by external system

• permission-denied: Permission denied

• service-unavailable: External system temporarily unavailable

• internal-error: Internal connector error

HTTP Status Mapping

Map errors to appropriate HTTP status codes:

• 400 Bad Request: Invalid configuration, invalid parameters

• 401 Unauthorized: Authentication failed (connector API key)

• 403 Forbidden: Permission denied

• 404 Not Found: Data source or resource not found

• 429 Too Many Requests: Rate limited

• 500 Internal Server Error: Internal connector error

• 503 Service Unavailable: External system unreachable or unavailable

Error Handling

In REST implementations, handle errors explicitly. Validate input, catch exceptions, and return appropriate HTTP status codes with error details in the standard error format. Distinguish between transient errors (network issues, rate limits) and permanent errors (invalid configuration) to return appropriate status codes.

Best Practices

Performance

Event Batching: Return multiple events per request (up to 5MB) rather than one at a time. This reduces the number of API calls.

Cursor Position Efficiency: Choose cursor positions that allow efficient querying. If your external system supports querying by timestamp or sequence number, use those.

Caching: Cache entity definitions and data source configurations. They don't change frequently.

Connection Pooling: Reuse HTTP connections to external systems. Use connection pooling libraries.

Rate Limiting: Respect rate limits of external systems. Implement backoff and retry logic.

Concurrent Requests: If your external system supports it, fetch multiple entity types concurrently.

Cursor Position Design

Stable Ordering: Ensure cursor positions provide stable, total ordering of events. If events can have the same timestamp, include an ID component.

Efficient Queries: Design cursor positions so you can efficiently query "events after position X" in your external system.

Initial Position: Always return a consistent initial position that comes before all events.

Position Gaps: Handle cases where events might be deleted or positions might have gaps. Your connector should handle "after position X" even if position X no longer exists.

Event Ordering

Strict Ordering: Events must be returned in ascending order by position. The last event in a response should have the highest position.

Consistency: If an entity is updated multiple times, return events in the order they occurred.

Deletes: Delete events should come after the last Upsert event for that entity.

Configuration Validation

Early Validation: Validate configuration as soon as it's received. Don't wait until events are fetched.

Connection Testing: If possible, test connectivity to the external system when configuration is set.

Clear Error Messages: Provide specific error messages indicating which fields are invalid and why.

Secret Fields: Mark sensitive fields (API keys, passwords) as secret: true in your config schema. These won't be displayed in Phonemos UI.

User Entity Handling

Email Matching: Use email addresses as the primary identifier for user entities. Phonemos matches users by email.

Complete User Data: Provide first_name, last_name, or display_name when available to help create Phonemos user accounts.

User References: Always reference users by their email address in entity references.

Error Recovery

Transient Errors: Distinguish between transient errors (network issues, rate limits) and permanent errors (invalid configuration). Retry transient errors.

Error Reporting: Return detailed error information in the details field to help users debug issues.

Status Endpoint: Use the status endpoint to report ongoing issues. If a data source is in an error state, return status: "Error" with details.

Testing

Unit Tests: Test your event conversion logic, cursor position serialization, and configuration parsing.

Integration Tests: Test against a real external system instance (or mock server) to verify API calls work correctly.

Edge Cases: Test with:

• Empty event streams

• Very large responses

• Invalid cursor positions

• Missing entities

• Deleted entities

• Configuration changes mid-sync

Sample Data: Create a sample connector with hardcoded data to test the Phonemos integration without needing an external system.

Security

API Key Validation: Always validate the X-Api-Key header. Use a strong, randomly generated secret.

Secret Storage: Store API keys and passwords securely. Don't log them.

HTTPS: Use HTTPS for all external system connections.

Input Validation: Validate all input from Phonemos and external systems. Don't trust external data.

Error Messages: Don't expose sensitive information (API keys, internal system details) in error messages.

Monitoring and Logging

Structured Logging: Use structured logging with appropriate log levels. Log data source operations, errors, and performance metrics.

Metrics: Consider adding metrics for:

• Request counts and durations

• Event counts per data source

• Error rates

• External system response times

Health Checks: Implement a health check endpoint (separate from the connector API) for monitoring.

Examples

Example External Data Connector

Conclusion

Building a custom connector for Phonemos involves:

1. Implementing the required endpoints: Info, data source management, events, status

2. Defining your entity schema: Fields, references, and types

3. Implementing event streaming: Converting external data to Phonemos events with cursor positions

4. Handling errors properly: Using standard error codes and HTTP status codes

5. Testing thoroughly: Unit tests, integration tests, and edge cases

External Data Connector Swagger Specification

Example External Data Connector

For questions or issues, consult the Phonemos documentation or contact the Phonemos team.