The term AI is everywhere and a central part of every marketing flyer. And while AI brings a lot of power and new capabilities, it is pretty complicated to get implementation right in order to fulfill the plethora of requirements from all different angles. To address these requirements, we offer a super flexible AI integration system that combines security, control and usability.

Why not just hard-code an LLM into the Phonemos platform?

Flexibility means complexity, so why are we not just delivering a fully integrated AI/LLM with our product like many other software vendors do?

We do this, because it is really hard to select the right system for your needs, once you consider all aspects:

• capabilities and fitness for purpose

• information security

• privacy & compliance needs

• economical cost

• ecological footprint

In the following sections, we will cover our considerations for each aspect, before we explain the solutions for your needs.

Model capabilities and fitness for purpose

While the quality of general purpose LLM are now leveling in their text processing capabilities, there are still some differences in language support, transparency on training method and data and built-in guardrails (“moral code” / ethics). If you have specific needs to select an LLM better addressing your niche of activities, Phonemos lets you do this by offering the choice of integration.

Most new generation LLM are now so powerful that they produce good results in most situations. Thus, capabilities start to become less relevant for choosing the right LLM.

Information security

Phonemos is built to store your intellectual property, trade secrets and knowledge that distinguish your organisation from your competition. You don’t want to feed your most important digital assets into just any system unless you verified, how they will be processed and protected. For example in some environments like government or banking, you might want to enforce that information is not crossing country borders and require localized solutions or even on premise LLM setups.

As not all organisations have the same needs in terms of security, we believe that the freedom of choice regarding the AI you trust your information is an important feature.

Privacy & compliance

Most organizations live in and need to respect a framework of specialized regulations like privacy (e.g. EU GDPR), banking (US SOX, DE BAFIN, CH FINMA, …) or technological regulations like the EU AI act. These regulations uphold the rights of individuals and often restrict cross-country data processing to some extent, unless certain provisions are met.

These regulations can make it difficult to use AI without closing both eyes while using them.

Economical cost

While AI can significantly boost your efficiency on certain tasks, it comes with a substantial cost as many standard operations on an LLM require a lot of processing power and generate substantial transaction costs that can be mostly ignored with low volume usage, but become significant under heavy use.

In addition to operational expenditure, conducting a security assessment on an AI solution needs a lot of time and effort, specialized knowledge etc. Thus, evaluating your LLM of choice and then using that exact system in your software tools like Phonemos just makes a lot more sense than doing the same assessment for every single tool.

Ecological footprint

The generation of a large language model requires a large amount of energy, water and hardware for processing power. The training of a single modern LLM model versions is estimated to cause 15’000 metric tons of CO2 emissions.

As most modern companies pledge to respect and preserve natural resources and need to live up to their promises, they need to be able to take these factors into consideration.

Choosing the right LLM for Phonemos

Phonemos can be made to work with most LLM. In order to do this, you opt-in into the necessary data transfers and you may enable or disable LLM use for each site. This means that if you have data that must not be transferred to an LLM, you may set up a separate site with LLM disabled while at the same time leveraging the power of LLMs in the other site.

LLM interface requirements

For semantic search, Phonemos needs an LLM that implements the Embeddings API (/v1/embeddings) with an OpenAI request/response format. Most vendors adopt this format even if the underlying model differs.

For chatbot-like scenarios, Phonemos needs an LLM that implements the Chat Completions API (/v1/chat/completions). Allmost all vendors now use this API.

For enterprise LLM integration scenarios, Phonemos comes with native support for the Model Context Protocol (MCP). With this interface, you can make information in Phonemos available to your enterprise LLM.

Default embeddings provider for cloud

In our cloud environment, we currently integrate by default with Mistral AI.

Implementing an enterprise search scenario

In early approaches to AI implementation, every software tool needed to choose its own LLM model and then build it into the software. This means, the AI capabilities live in the knowledge silos of each software application. This is now changing with the Model Context Protocol (MCP) being adopted by most large vendors.

With MCP, you can plug-in an external data source like Phonemos into an LLM and then let the LLM perform tasks with Phonemos. You may combine this with other MCP data sources to let LLMs perform tasks across applications.