Skip links

The ThoughtForma Approach to Knowledge: Leveraging Serverless Technology for Data Mastery

We’re fond of saying that one of the most underutilized assets many businesses possess is their own data. Often, the potential value within this data remains untapped due to a lack of awareness, or the perceived complexity of accessing and unlocking its value. At ThoughtForma, our vision is to simplify the way businesses leverage data and AI to build powerful conversational applications. We strive to use technology to get technology out of people’s way. This article explores one key dimension of our knowledge handling approach – our serverless knowledge graph – to illustrate system architecture done the ThoughtForma ‘way’.

The ThoughtForma Approach to Knowledge

ThoughtForma’s knowledge architecture comprises multiple components that work together to maximize the value extracted from data. These components include:

  1. Knowledge Graph: Built on AWS DynamoDB and maintained with AWS Lambda functions, our knowledge graph links diverse information sources through adjacency lists, creating a rich, interconnected data structure.
  2. Knowledge Base: Using a serverless vector store (one per ThoughtForma application) backed by S3, we ensure fast and efficient retrieval of relevant information through vector search techniques.
  3. Dynamic Structured Data Repositories: These repositories, also using AWS DynamoDB, store structured data in a flexible, scalable format, allowing for dynamic querying and updating.
  4. Data Summarization and Analysis: Upon ingestion, we apply techniques to analyze and summarize data, ensuring that it is concise, relevant, and ready for immediate use.

Building a Serverless Knowledge Graph

AWS DynamoDB as the Datastore

Instead of using a traditional graph database, we leverage AWS DynamoDB as the datastore for our knowledge graph. Here’s why we chose this approach:

  • Overkill Avoidance: Traditional graph databases offer extensive features that can be overkill for our needs. Typically, three hops are enough for our queries, making DynamoDB a suitable choice.
  • Single Table Design with Adjacency Lists: We use a single DynamoDB table to store nodes and edges, linked via adjacency lists. This design allows us to efficiently query relationships and traverse the graph.
  • Source Records: Each node and edge record is linked to the original source of knowledge, ensuring traceability and data integrity.

Maintenance with AWS Lambda Functions and Step Functions

Our knowledge graph is maintained using AWS Lambda functions and Step Functions, which provide serverless compute power to:

  • Update Nodes and Edges: Lambda functions are triggered to update nodes and edges based on new data ingested into the system.
  • Link Source Records: They ensure that each piece of data is appropriately linked to its source, maintaining the graph’s accuracy and relevance.
  • Handle Disambiguation: Specialized functions work to resolve ambiguities in data, ensuring that nodes represent distinct entities accurately.

Specialized Language Processing Models

For entity resolution and relationship extraction, we use specialized custom language processing models running on serverless GPU ‘lambda-like’ functions. These models:

  • Extract Entities and Relationships: Efficiently extract meaningful entities and relationships from unstructured data.
  • Populate the Knowledge Graph: Feed this extracted information into our DynamoDB-backed knowledge graph, ensuring it stays rich and up-to-date.

Why Serverless?

Cost Efficiency

Serverless architectures inherently reduce costs by eliminating the need for server management and allowing us to pay only for what we use. This cost efficiency extends to our users, enabling them to build AI-native apps without incurring high infrastructure costs.

Scalability

With serverless technology, we can scale seamlessly to handle varying data loads. AWS Lambda and DynamoDB automatically scale to accommodate spikes in data processing and querying, ensuring consistent performance. This scalability benefits our users by providing a robust platform that adapts to their needs. When we sign up a new customer and they immediately set about creating 100 new AI assistants, this is when our scaleability begins to shine.

Speed to Market

By leveraging serverless components, we can develop and deploy features faster. This agility allows us to quickly respond to user needs and integrate new capabilities into ThoughtForma, helping our users bring their AI-native apps to market faster.

The Knowledge Graph as Part of a RAG System

The serverless knowledge graph is a crucial part of our Retrieval-Augmented Generation (RAG) system, which complements our vector store and structured data handling. Here’s how it integrates and adds value:

Enhancing the Knowledge Base

The knowledge graph enriches our vector store by providing a structured, interconnected view of the data. This enhances the context and relevance of information retrieved by our RAG system.

Driving AI Intuition

By organizing data into a knowledge graph, ThoughtForma can intuit what users might want their apps to do based on the data they upload and ingest. This AI-driven intuition helps users unlock the full potential of their data effortlessly. Why spend time designing complex applications? Our platform can take a look at your data and suggest (and provide) appropriate capabilities automatically.

Simplifying User Experience

Our goal is to make technology invisible to the user. By automating complex data processing and linking tasks, we ensure that users can focus on their core business objectives without getting bogged down by technical details.

Benefits for ThoughtForma Users

Reducing Costs

Serverless computing minimizes infrastructure costs, allowing us to pass on savings to our users.

Increasing Speed to Market

Rapid development and deployment cycles mean faster time-to-value for our customers.

Enhancing Ease of Use

ThoughtForma’s intuitive interface and automated data handling make advanced AI accessible to everyone, regardless of technical expertise.

Conclusion

At ThoughtForma, we believe in the transformative power of data and AI. Our serverless knowledge graph, built using AWS DynamoDB, Lambda, Step Functions, and cutting-edge language processing models, is an example of our commitment to innovation. By reducing costs, increasing speed to market, and simplifying the user experience, we’re helping businesses unlock the hidden value in their data and stay ahead in today’s competitive landscape.

Join us on this journey and discover how ThoughtForma can revolutionize the way you harness data and AI. Let’s build the future, together.

Leave a comment

🍪 This website uses cookies to improve your web experience.