# Rememberizer LLM Ready Documentation
*Generated at 2025-01-16 19:10:30 PST. Available as raw content at [Rememberizer llms-full.txt](https://llm.rememberizer.ai/llms-full.txt).*
This document provides a comprehensive, consolidated reference of Rememberizer's documentation, optimized for large language model (LLM) consumption. It combines various documentation sources into a single, easily accessible format, facilitating efficient information retrieval and processing by AI systems.
```
==> SUMMARY.md <==
# Table of contents
* [Why Rememberizer?](README.md)
* [Background](background/README.md)
* [What are Vector Embeddings and Vector Databases?](background/what-are-vector-embeddings-and-vector-databases.md)
* [Personal](personal/README.md)
* [Rememberizer Slack integration](personal/rememberizer-slack-integration.md)
* [Rememberizer Dropbox integration](personal/rememberizer-dropbox-integration.md)
* [Rememberizer Google Drive integration](personal/rememberizer-google-drive-integration.md)
* [Rememberizer Gmail integration](personal/rememberizer-gmail-integration.md)
* [Rememberizer Memory integration](personal/rememberizer-memory-integration.md)
* [Rememberizer App](personal/rememberizer-app.md)
* [Mementos Filter Access](personal/mementos-filter-access.md)
* [Manage third-party apps](personal/manage-third-party-apps.md)
* [Common knowledge](personal/common-knowledge.md)
* [Search your knowledge](personal/search-your-knowledge.md)
* [Manage your embedded knowledge](personal/manage-your-embedded-knowledge.md)
* [Developer](developer/README.md)
* [Registering Rememberizer apps](developer/registering-rememberizer-apps.md)
* [Authorizing Rememberizer apps](developer/authorizing-rememberizer-apps.md)
* [Creating a Rememberizer GPT](developer/creating-a-rememberizer-gpt.md)
* [Talk-to-Slack the Sample Web App](developer/talk-to-slack-the-sample-web-app.md)
* [Registering and using API Keys](developer/registering-and-using-api-keys.md)
* [LangChain integration](developer/langchain-integration.md)
* [Vector Stores](developer/vector-stores.md)
* [API documentations](developer/api-documentations/README.md)
* [Memorize content to Rememberizer](developer/api-documentations/memorize-content-to-rememberizer.md)
* [Retrieve documents](developer/api-documentations/retrieve-documents.md)
* [Retrieve document contents](developer/api-documentations/retrieve-document-contents.md)
* [Retrieve Slack's content](developer/api-documentations/retrieve-slacks-content.md)
* [Search for documents by semantic similarity](developer/api-documentations/search-for-documents-by-semantic-similarity.md)
* [List available data source integrations](developer/api-documentations/list-available-data-source-integrations.md)
* [Retrieve current user's account details](developer/api-documentations/retrieve-current-users-account-details.md)
* [Get all added public knowledge](developer/api-documentations/get-all-added-public-knowledge.md)
* [Vector Store APIs](developer/api-documentations/vector-store/README.md)
* [Get vector store's information](developer/api-documentations/vector-store/get-vector-stores-information.md)
* [Get a list of documents in a Vector Store](developer/api-documentations/vector-store/get-a-list-of-documents-in-a-vector-store.md)
* [Get the information of a document](developer/api-documentations/vector-store/get-the-information-of-a-document.md)
* [Add new text document to a Vector Store](developer/api-documentations/vector-store/add-new-text-document-to-a-vector-store.md)
* [Upload files to a Vector Store](developer/api-documentations/vector-store/upload-files-to-a-vector-store.md)
* [Update file's content in a Vector Store](developer/api-documentations/vector-store/update-files-content-in-a-vector-store.md)
* [Remove a document in Vector Store](developer/api-documentations/vector-store/remove-a-document-in-vector-store.md)
* [Search for Vector Store documents by semantic similarity](developer/api-documentations/vector-store/search-for-vector-store-documents-by-semantic-similarity.md)
* [Notices](notices/README.md)
* [Terms of Use](notices/terms-of-use.md)
* [Privacy Policy](notices/privacy-policy.md)
* [B2B](notices/b2b/README.md)
* [About Reddit Agent](notices/b2b/about-reddit-agent.md)
* [Releases](releases/README.md)
* [Jan 17th, 2025](releases/jan-17th-2025.md)
* [Dec 27th, 2024](releases/dec-27th-2024.md)
* [Dec 20th, 2024](releases/dec-20th-2024.md)
* [Dec 13th, 2024](releases/dec-13th-2024.md)
* [Dec 6th, 2024](releases/dec-6th-2024.md)
* [Nov 29th, 2024](releases/nov-29th-2024.md)
* [Nov 22nd, 2024](releases/nov-22nd-2024.md)
* [Nov 15th, 2024](releases/nov-15th-2024.md)
* [Nov 8th, 2024](releases/nov-8th-2024.md)
* [Nov 1st, 2024](releases/nov-1st-2024.md)
* [Oct 25th, 2024](releases/oct-25th-2024.md)
* [Oct 18th, 2024](releases/oct-18th-2024.md)
* [Oct 11th, 2024](releases/oct-11th-2024.md)
* [Oct 4th, 2024](releases/oct-4th-2024.md)
* [Sep 27th, 2024](releases/sep-27th-2024.md)
* [Sep 20th, 2024](releases/sep-20th-2024.md)
* [Sep 13th, 2024](releases/sep-13th-2024.md)
* [Aug 16th, 2024](releases/aug-16th-2024.md)
* [Aug 9th, 2024](releases/aug-9th-2024.md)
* [Aug 2nd, 2024](releases/aug-2nd-2024.md)
* [Jul 26th, 2024](releases/jul-26th-2024.md)
* [Jul 12th, 2024](releases/jul-12th-2024.md)
* [Jun 28th, 2024](releases/jun-28th-2024.md)
* [Jun 14th, 2024](releases/jun-14th-2024.md)
* [May 31st, 2024](releases/may-31st-2024.md)
* [May 17th, 2024](releases/may-17th-2024.md)
* [May 10th, 2024](releases/may-10th-2024.md)
* [Apr 26th, 2024](releases/apr-26th-2024.md)
* [Apr 19th, 2024](releases/apr-19th-2024.md)
* [Apr 12th, 2024](releases/apr-12th-2024.md)
* [Apr 5th, 2024](releases/apr-5th-2024.md)
* [Mar 25th, 2024](releases/mar-25th-2024.md)
* [Mar 18th, 2024](releases/mar-18th-2024.md)
* [Mar 11th, 2024](releases/mar-11th-2024.md)
* [Mar 4th, 2024](releases/mar-4th-2024.md)
* [Feb 26th, 2024](releases/feb-26th-2024.md)
* [Feb 19th, 2024](releases/feb-19th-2024.md)
* [Feb 12th, 2024](releases/feb-12th-2024.md)
* [Feb 5th, 2024](releases/feb-5th-2024.md)
* [Jan 29th, 2024](releases/jan-29th-2024.md)
* [Jan 22nd, 2024](releases/jan-22nd-2024.md)
* [Jan 15th, 2024](releases/jan-15th-2024.md)
* [Rememberizer LLM Ready Documentation](rememberizer-llm-ready-documentation.md)
==> README.md <==
---
description: Introduction
---
# Why Rememberizer?
Generative AI apps work better when they have access to background information. They need to know what you know. A great way to achieve that is to give them access to relevant content from the documents, data and discussions you create and use. That is what Rememberizer does.
==> personal/rememberizer-slack-integration.md <==
---
description: >-
This guide will walk you through the process of integrating your Slack
workspace into Rememberizer as a knowledge source.
---
# Rememberizer Slack integration
1. Sign in to your account.
2. Navigate to **Personal > Your Knowledge** tab, or visit [https://rememberizer.ai/personal/knowledge](https://rememberizer.ai/personal/knowledge). You should see all available knowledge sources there, including Slack.
Your Knowledge, ready to connect to Slack
3. Click the **"Connect"** button of the Slack knowledge source. You will be redirected to a new page asking for your permission to allow Rememberizer to access your Slack workspace.
Slack OAuth screen
> **Note:** If you see a warning that this application is not authorized by Slack, it is because Rememberizer is intended to search for Slack content outside of Slack, which is against the [Slack App Directory Guidelines](https://api.slack.com/directory/guidelines).
4. Click **"Allow"** to install the Rememberizer Slack app to your workspace. Once you've granted the necessary permissions, you'll be redirected back to our platform, at which you should see your Slack workspace connected and a side panel pops up.
A-COMPANY has been added as a knowledge source
5. Now that you're connected, you need to specify which channels our product should source messages from. Choose your desired files or folders from the side panel. If the side panel does not appear, click on the **"Select"** button to open the side panel.
Select channels to be embedded as knowledge
6. After selecting your channels, our system will begin embedding the messages and files. This process may take a few minutes depending on the amount of data.
### What's next?
Use the [Memento](mementos-filter-access.md) feature to filter access to the sourced data. Combine this with your knowledge from other applications such as Google Drive, Box, Dropbox, etc. to form a comprehensive memento.
You can also [Search Your Knowledge](https://rememberizer.ai/personal/search) through our web UI, or better, use this knowledge in an LLM through our GPT app or our public API.
And that's it! If you encounter any issues during the process, feel free to contact our support team.
==> personal/manage-third-party-apps.md <==
# Manage third-party apps
## Explore third-party apps and service
User can view and explore all third-party apps that connect with Rememberizer in **App directory** page with the below instructions.
* On the navigation bar, choose **Personal > Find an App**. Then, you will see the App directory page.
Navigation bar browsing App Directory page
App directory page
* Find the app you want to explore. You can do this by type the name of the app in search bar with optional **filter and sorting order.**
Search bar with filter and sort order button
* Click on the **name of the third-party app** or **Explore button** to open the app.
App's name and Explore button
* When using the app, it will requires authorizing the apps with Rememberizer. Technical details of the flow can be visited at [authorizing-rememberizer-apps.md](../developer/authorizing-rememberizer-apps.md "mention") page. We will use **Rememberizer GPT app** as an example of the UI flows of authorization. After the first chat, you will see the app ask to sign in the Rememberizer.
Sign in request from Rememberizer GPT app
* Click on the **Sign in** button. You will be redirected to the Authorization page.
Authoriztion page
* You can modify the Memento and Memory that the app can view and use by click on the **Change** button and select what you want.
> **Note:** Detail information about Memento, please visit [mementos-filter-access.md](mementos-filter-access.md "mention") page.
> **Note:** Detail information about Memory integration, please visit [rememberizer-memory-integration.md](rememberizer-memory-integration.md "mention") page.
* Click **Authorize** to complete the process. You then will be directed back to the app again and you can chat with it normally.
> **Note:** In case you click **Cancel** button, you will be directed to the app landing page again and the app will not be displayed in the **App directory** page but will instead be on **Your connected apps** page. More detail information please visit second part [#manage-your-connected-apps](manage-third-party-apps.md#manage-your-connected-apps "mention") if you want to completely cancel the authorization process.
Success connected account
## Manage your connected apps
On the **App directory** page, choose **Your connected apps** to browse the page.
Your connected apps page
This page categorizes apps into two types based on their status: **Pending Apps** and **Connected Apps**.
* **Pending Apps**: These are the apps which you click **Cancel** button while authorizing the the app on Rememberizer.
* Click **Continue** if you want to complete the authorization process.
* Otherwise, click **Cancel** to completely withdraw the authorization. The app will then be displayed in **App Directory** page again.
* **Connected Apps:** You can config the **Memento** or **Memory integration** of specific connected app by click on the Change option (or Select if the Memento has not been chosen). Click **Disconnect** if you want to disconnect the third-party app from the Rememberizer.
==> personal/rememberizer-memory-integration.md <==
# Rememberizer Memory integration
### Introduction
Rememberizer Memory allows third party apps to store and access data in a user's Rememberizer account, providing a simple way for valuable information to be saved and utilize across multiple user' applications.
### Benefits
#### For User
Shared Memory creates a single place where key results and information from all the user's apps are available in one location. Some benefits for user include:
* Easy Access: Important data is centralized, allowing both the user and their apps to easily access results from multiple apps in one place.
* Sync Between Apps: Information can be shared and synced between a user's different apps seamlessly without extra effort from the user.
* Persistent Storage: Data remains accessible even if individual apps are uninstalled, unlike app-specific local storage.
#### For App Developers
The Shared Memory provides app developers a simple way to access data from a user's other connected apps:
* No Backend Needed: Apps do not need to develop their own custom backend systems to store and share data.
* Leverage Other Apps: Apps can build on and utilize public data generated by a user's other installed apps, enriching their own functionality.
* Cross-App Integration: Seamless integration and data sharing capabilities are enabled between an app developer's different apps.
By default all apps have read-only access to Shared Memory, while each app can write only to its own memory space. The user has controls to customize access permissions as needed. This balances data sharing with user privacy and control.
### Config Your Memory
#### Global Settings
The Global Settings allow user to configure the default permissions for all apps using Shared Memory. This includes:
Config Memory in Knowledge Page
#### Default Memory and Data Access Permissions for Apps
* **Read Own/Write Own:** Apps are exclusively permitted to access and modify their own memory data.
* **Read All/Write Own:** Apps can read memory data across all apps but are restricted to modifying only their own memory data.
* **Disable Memory:** By default, apps cannot access or store memory data.
* **Apply to All Option**: User can apply all app-specific permission settings back to the defaults chosen in Global Settings.
User can clear all Memory documents with _**Forget your memory**_ option:
Confirmation Modal when Forget Memory
#### App Settings
For each connected app, user can customize the Shared Memory permissions. Click on the **"Find an App"**, then click **"Your connected apps"** or go to the link [https://rememberizer.ai/personal/apps/connected](https://rememberizer.ai/personal/apps/connected) to see the list of your connected apps. Then, click **"Change"** on the Memory of the app you want to custom:
Config Memory for each App in Connected Apps Page
#### Memory Access Permissions for Apps
* **Read Own/Write Own**: Permissions allow the app to only access and modify its own memory data, preventing it from interacting with other apps' memory.
* **Read All/Write Own**: The app can view memory data from all apps but is restricted to modifying only its own memory data.
* **Disable Memory**: The app is prohibited from accessing or modifying memory data.
This allows user fine-grained control over how each app can utilize Shared Memory based on the user's trust in that specific app. Permissions for individual apps can be more restrictive than the global defaults.
Together, the Global and App Settings give user powerful yet easy-to-use controls over how their data is shared through Shared Memory.
### Integrate with Memory Feature
#### API Endpoint
Rememberizer expose an API end point [/**api/v1/documents/memorize/**](https://docs.rememberizer.ai/\~/changes/8nxu1gB5bGpm7B5IZlQ8/developer/api-documentations/memorize-content-to-rememberizer) to let GPT App call to memorize the content.
Note: This api is available for Memory with [3rd-party apps with OAuth2 authentication](../developer/authorizing-rememberizer-apps.md) only (not [API-key](../developer/registering-and-using-api-keys.md) yet).
#### Memorize your knowledge
After authorizing with Rememberizer, the third party app can memorize it valuable knowledge.
Here, we will demonstrate a process using Remembeizer GPT App.
* After using Rememberizer GPT App, user want to memorize the third point "Zero-Cost Abstractions".
Interacting with Rememberizer GPT Apps
* To use the Rememberizer App's Memory feature, user must first authorize the app to access your project. Use the **memorize** command to tell the app what knowledge it needs to store.
Sign In to authorize Rememberizer
* User can Config the Memory Option here, with the default value is based on the Global Config
Authorizing Screen
The Rememberizer now successfully memorizes knowledge.
* In Rememberizer, user can see the recent content at **Embed Knowledge Details** page.
With the **Talk to Slack** app, user can seamlessly apply and continue their progress using the data they have committed to memory. For example, memorized information they can easily query and retrieve
Recall Memory Data in another app
### Using Memory Data via Memento
* Another way to utilize the memory data is by creating **Memento** and refine the Memory into it. Visit [Memento Feature](mementos-filter-access.md#how-to-create-a-mementos) section for further information about creation instruction.
* Rememberizer saves content into files and user can choose any app to refine its content into **Memento**.
> Note: In older version, Rememberizer saves content into files and combine into folder for each date.
With the [Memento Feature](https://docs.rememberizer.ai/personal/mementos-filter-access#what-is-a-memento-and-why-do-they-exist), user can utilize the Memory data even when the Memory App Config is Off.
### Search Memory document in Rememberizer
You can also [Search Your Knowledge](https://rememberizer.ai/personal/search) through our web UI, or better, use this knowledge in an LLM through our GPT app or our public API.
==> personal/rememberizer-dropbox-integration.md <==
---
description: >-
This guide will walk you through the process of integrating your Dropbox into
Rememberizer as a knowledge source.
---
# Rememberizer Dropbox integration
1. Sign in to your account.
2. Navigate to **Personal > Your Knowledge** tab, or visit [https://rememberizer.ai/personal/knowledge](https://rememberizer.ai/personal/knowledge). You should see all available knowledge sources there, including Dropbox.
Dropbox, ready to be connected as a knowledge source
3. Click **"Allow"** to install the Rememberizer Dropbox app to your account.
4. Once you've granted the necessary permissions, you'll be redirected back to our platform, at which you should see your Dropbox account connected and a side panel pops up.
5. Now that you're connected, you need to specify which files and folders our product should embed. Choose your desired files or folders from the side panel. If the side panel does not appear, click on the **"Select"** button to open the side panel.
7. After selecting your files and folders, click **"Add"** and our system will begin the embedding. This process may take a few minutes depending on the amount of data.
### Connecting to another Dropbox account
If you disconnect your Dropbox knowledge and then reconnect, chances are Dropbox will automatically connect to your previous Dropbox account, skipping the Authorize screen completely.
If you want to connect using a different Dropbox account:
1. Go to the Dropbox website and sign in with your previous account credentials.
2. Once you're logged in, click on your profile picture in the upper-right corner.
3. Select "Settings" from the dropdown menu.
4. In the settings menu, select the "Connected apps" tab.
5. Find the Rememberizer app in the list of connected apps and click "Disconnect" next to it.
6. Sign out of your previous Dropbox account.
7. Now, when you try to connect Dropbox to the Rememberizer app again, it will prompt you to authorize a new Dropbox account.
### What's next?
Use the [Memento](mementos-filter-access.md) feature to filter access to the sourced data. Combine this with your knowledge from other applications such as Google Drive, Slack, etc. to form a comprehensive memento.
You can also [Search Your Knowledge](https://rememberizer.ai/personal/search) through our web UI, or better, use this knowledge in an LLM through our GPT app or our public API.
And that's it! If you encounter any issues during the process, feel free to contact our support team.
==> personal/rememberizer-google-drive-integration.md <==
---
description: >-
This guide will walk you through the process of integrating your Google Drive
into Rememberizer as a knowledge source.
---
# Rememberizer Google Drive integration
1. Sign in to your account.
2. Navigate to **Personal > Your Knowledge** tab, or visit [https://rememberizer.ai/personal/knowledge](https://rememberizer.ai/personal/knowledge). You should see all available knowledge sources there, including Google Drive.
3. Click the **"Connect"** button of the Google Drive knowledge source. You will be redirected to a new page asking for your permission to allow Rememberizer to access your Google Drive. Select your Google Drive account.
4. Approve the app by clicking on "**Continue"**.
5. Allow Rememberizer to **See and download all your Google Drive files** by clicking **"Continue".**
6. You'll be redirected back to our platform, at which you should see your Drive account connected and a side panel pops up.
7. Now that you're connected, you need to specify which files and folders our product should embed. Choose your desired files or folders from the side panel. If the side panel does not appear, click on the **"Select"** button to open the side panel.
8. After selecting the files, click **"Add"** to start embedding your knowledge. You also need to check the box to agree with Rememberizer's policy of sharing your Google Drive data with third-party applications that you have specifically approved.
9. After selecting your files and folders, our system will begin embedding the messages and files. This process may take a few minutes depending on the amount of data.
### Limitations of Google Drive Integration
* While Rememberizer's Google Drive integration allows you to access and embed a wide range of files and folders, it currently cannot access or embed files from the "Computers" section, which is used for backing up files from your computer due to restrictions set by Google.
* If you need to embed files from your local computer, we recommend using our Rememberizer Agent desktop application. To learn more about the Rememberizer Agent and how to install and use it, please refer to our [Rememberizer Agent](rememberizer-app.md) guide.
### What's next?
Use the [Memento](mementos-filter-access.md) feature to filter access to the sourced data. Combine this with your knowledge from other applications such as Slack, Box, Dropbox, etc. to form a comprehensive memento.
You can also [Search Your Knowledge](https://rememberizer.ai/personal/search) through our web UI, or better, use this knowledge in an LLM through our GPT app or our public API.
And that's it! If you encounter any issues during the process, feel free to contact our support team.
==> personal/README.md <==
# Personal
==> personal/search-your-knowledge.md <==
---
description: >-
In Rememberizer, you can post a theme or question, and Rememberizer will
provide a list of files and extracts parts that are conceptually similar.
---
# Search your knowledge
## Search in Rememberizer
* In the navigation bar, choose **Personal > Search Your Knowledge**. Then you will see the search page in Rememberizer
* Type the question or theme you want to search, then choose the the memento you want to limit the app's access and click Rememberizer button (or press Enter). The search process may take a few minutes, depending on the amount of data in the Memento.
Memento Filtering in search Rememberizer
* Eventually, you will see list of documents matching question or theme you require. You can click to the file and it will dropdown the matching chunk text related to your question or theme.
An example of search result
==> personal/rememberizer-app.md <==
# Rememberizer App
### Introduction.
Rememberizer App is a MacOS desktop application that converts your local files into vector embeddings and uploads them to your Rememberizer's knowledge as a data source. The application enables other LLMs to query your embeddings through Rememberizer's APIs to make answers based on your local files content.
### Benefits.
* **Data Utilization:** The application helps you to utilize your local files in a meaningful and productive way. It extracts valuable data from your files and makes it available for machine learning processes.
* **Ease of Use:** The application has a user-friendly interface and it’s easy to install and use. It does all the heavy lifting of converting files and uploading the data, so you don't have to.
* **Integration:** The Rememberizer App seamlessly integrates with other LLMs. This allows them to query your embeddings through Rememberizer's APIs to generate answers based on the content of your local files.
### Installation.
1. Download a version of Rememberizer App from [the links provided here](rememberizer-app.md#download-links).
2. Once the download is complete, locate the .dmg file in your downloads folder and double click on it.
3. Drag the Rememberizer App into your Applications folder when the new window opens.
4. Go to your Applications folder and click on the Rememberizer App to open it.
### How to Use.
1. **Sign In:** To use the Rememberizer App, you need to sign in using your Rememberizer account. If you do not have a Rememberizer account, you'll need to create one.
2. **Add Folders to the Data Source:** Once you have signed in, the Rememberizer App will begin operating in the background. You can access it by clicking the small icon located in the status bar, as illustrated below. For first-time use, you will need to add folders to the data source. This enables the Rememberizer App to convert files within those folders into vector embeddings and upload them to your Rememberizer Knowledge.
3. **Embedding And Uploading:** The software will seamlessly integrate these vector embeddings into your Rememberizer Knowledge database. Track the upload process via the Rememberizer status tab.
### Download links.
* Rememberizer App 1.6.1 ([MacOS](https://www.dropbox.com/scl/fi/hzytquytxmuhpov67spru/rememberizer-app-1.6.1.dmg?rlkey=0p30ok9qt4e33ua8scomagzev\&st=8yys88d5\&dl=1)) - [Release note](rememberizer-app.md#version-1.3.1-july-25-2024).
Please note, it is recommended to always use the latest version of the software to take advantage of all the latest features and improvements.
Rememberizer App is designed to make the process of converting local files to vector embeddings as seamless as possible. Enjoy using your data in a more productive way!
### Release notes.
### Version 1.6.1 (October 4th 2024)
#### Features and Improvements
* **Support for Empty Folders**: Users can now add empty folders as a data source.
* **Minor Improvements**: Enhancements to the user interface and performance.
* **GPU Support and Performance Improvements**: Added support for GPU acceleration to enhance processing speed.
* **Enhanced Embedding Program**: Configured to support the MPS version of PyTorch, optimizing for machine-specific builds.
* **Intelligent CPU Detection**: Implemented detection of CPU type to ensure the most suitable version of the embedding program is used.
* **Improved Data Source Management**: Utilized the Batch Delete API for efficient file deletion in removed data sources.
* **Support for All Plain Text Files**: Enabled processing of various plain text file types.
* **Adherence to Gitignore Rules**: Files ignored by gitignore rules in Git repositories are now excluded from processing.
==> personal/common-knowledge.md <==
---
description: >-
Enhance your knowledge or get started fast by adding AI access to pre-indexed
data from us and others.
---
# Common knowledge
## What is common knowledge
In Rememberizer, registered users **(publishers)** can select their uploaded documents through mementos and share them publicly as common knowledge. Other users **(subscribers)** can access this public knowledge and add it to their own resources.
By contributing their data, other users can collectively enhance the available information on the common knowledge page. This collaborative approach allows all users to access a richer data source, thereby improving the learning capabilities of their AI applications.
## Add public common knowledge
In order to subscribe a common knowledge to your resource, follow the instructions below
* On navigation bar, choose **Personal > Common Knowledge**. Then, you will see the public common knowledge page.
* Then, look for the common knowledge you want to subscribe. You can look up the knowledge by typing the knowledge's name on search bar. You can optionally choose the filter option next to the search bar.
Filter of search bar
Example of a search result
* Then click **Add** button on the public common knowledge. After successful subscribe, you will see the **Add** button change to **Remove** button.
Unadded common knowledge
Added common knowledge
* Later, if you want to remove a subscribed knowledge, click the **Remove** button.
## Create a common knowledge
For detailed instructions of creating and sharing a common knowledge, visit this page [registering-and-using-api-keys.md](../developer/registering-and-using-api-keys.md "mention").
==> personal/mementos-filter-access.md <==
---
description: Use a Memento with each app integration to limit its access to your Knowledge
---
# Mementos Filter Access
### What is a Memento and Why do they Exist?
A major purpose of Rememberizer is to share highly relevant extracts of your data with 3rd party applications in a controlled fashion. This is achieved through the application of a single **Memento** to each application that is integrated with Rememberizer that you also choose to authorize to access your data in Rememberizer.
The current implementation of Memento allows the user to select specific files, documents or groups of content such as a folder or channel that can be used by that application. Later implementations will add additional ways to filter 3rd party access such as time frames like "created in the last 30 days".\
\
Two default values are "None" and "All". All shares every file that the user has allowed Rememberizer to access. None shares nothing with the app in question. Selecting None allows a user to select an app and integrate it with Rememberizer without having to decide then and there what content to make available. Selecting a Memento with None or editing an existing applied Memento to share None is a way to turn off an apps access to user data without having to remove the integration. This is like an off switch for your data. Custom Mementos can be purpose made and have names that reflect that, such as "Homework" or "Marketing".
### How to create a Mementos?
This guide will walk you through the process of creating a Mementos
1. Navigate to **Personal > Memento: Limit Access** in tab, or visit [https://rememberizer.ai/personal/memento](https://rememberizer.ai/personal/memento). You should see all Mementos the left of the screen
2. Click **Create a new memento**. Then fill the name for your custom memento and click **Create**. After that, you should your memento added and list of data sources can be included in your memento.
3. Click **Refine** on the data source you want to refine, the side panel will pop up. Then choose to add folders or files, and click **Refine** to add those data sources to the Memento.
4. Also, for common knowledge source, you can click **Add** to include the knowledge in Memento.
==> personal/manage-your-embedded-knowledge.md <==
---
description: >-
Rememberizer allows users to efficiently manage their stored files from
various sources. This section will show you how to access, search, filter and
manage your uploaded file in the knowledge
---
# Manage your embedded knowledge
## Browse Embedded Knowledge Details page
On the navigation bar, choose **Personal > Your Knowledge**. Locate the **View Details** button on the right side of the "Your Knowledge" section and click it. Then, you will see the **Embedded knowledge detail** page.
Your Knowledge section and View Details button
Embed Knowledge Detail page
The table of knowledge files' details includes these attributes:
* **Documents:** Name of the document or slack channel.
* **Source:** The resource from where the file is uploaded (Drive, Mail, Slack, Dropbox, and Rememberizer App).
* **Directory:** The directory where the file locates in the source.
* **Status**: The status of the file (indexing, indexed or error).
* **Size**: The size of the file.
* **Indexed on**: The date when the file is indexed.
* **Actions:** The button to delete the file. For file whose status is error, there will also be a retry icon next to the trash icon (delete button).
## Features of detail page
### Search and filter the files
User can search the document by name with the **search bar**. Type the name in the bar, then press Enter to get your result.
Result of a search
You can also optionally choose the **Status filter** and **Source filter.** This will quickly locate specific documents by narrowing down your search criteria.
Source filter
Status filter
### Delete an uploaded file
Find the file you want to delete (by search if needed). Then click on the **trash icon** on the **Action** column.
File with delete icon
A modal will pop up to confirm deletion. Click **Confirm** then you will see the file deleted.
Delete confirmation modal
### Retry indexing error files
User can retry to embed those files which Rememberizer failed to index. To retry indexing a specific file, simply click the retry button next to the delete button in **Action** column.
Retry button for specific error file
If user want to retry indexing all error files, click the retry button next to the label of **Action** column.
Retry button for all error files
Below is the image after sucessful retry indexing the error file from Gmail integration.
In their most advanced form (as used by Rememberizer) vector embeddings are created by language models with architectures similar to the AI LLMs (Large Language Models) that underpin OpenAI's gpt models and ChatGPT service as well as models/services from Google (Gemini) , Anthropic (Claude), Facebook (LLama 2) and others. For this reason it is natural to use vector embeddings to discover relevant knowledge to include in the context of AI model prompts. The technologies are complementary and somewhat equivalent. For this reason most providers of LLMs as a service will also produce vector embeddings as a service (for example: [a blog from Together AI](https://www.together.ai/blog/embeddings-endpoint-release) or [another blog from OpenAI](https://openai.com/blog/introducing-text-and-code-embeddings)).
What does a vector embedding look like? Consider a co-ordinate (x,y) in two dimensions. If it represents a line from the orgin to this point, we can think of it as a line with a direction, in other words a _vector in two dimensions._ In our context, a vector embedding will be a list of something like 768 numbers representing a vector in a 768-dimensional space. Ultimately this list of numbers can represent weights between zero and one in a Transformer model that define the meaning in a phrase such as "A bolt of lightening out of the blue." This is fundamentally the same underlying representation of meaning used in GPT-4 for example. As a result, we can expect a good vector embedding to enable the same brilliant apparent understanding that we see in modern AI language models.
\
It is worth noting that vector embeddings can be used to represent more than just text, but also other types of data such as images or sound. And with a properly trained model one can compare across media, so that a vector embedding on a block of text can be compared to an image, or _visa versa_. Today Rememberizer enables searches within just the text component of user documents and knowledge. But text-to-image and image-to-text search are on the roadmap.\
\
Google uses vector embeddings to power their text search (text-to-text) and also their image search (text-to-image) ([reference](https://cloud.google.com/blog/topics/developers-practitioners/meet-ais-multitool-vector-embeddings)) . Facebook has contemplated using embeddings for their social network search ([reference](https://research.facebook.com/publications/embedding-based-retrieval-in-facebook-search/)). Snapchat uses vector embeddings to understand context in order to serve the right ad to the right use at the right time ([reference](https://eng.snap.com/machine-learning-snap-ad-ranking)).
To deeply understand how vector embedding and vector databases work start with the [overview](https://huggingface.co/blog/getting-started-with-embeddings) from Hugging Face. Pinecone (a vector embedding database as a service) has a good [overview](https://www.pinecone.io/learn/vector-embeddings/) as well.
Another great source for understanding the search and knowledge in vectors is the Meta/Facebook paper and code for the FAISS library. "FAISS: A Library for Efficient Similarity Search and Clustering of Dense Vectors" by Johnson, Douze, and Jégou (2017): FAISS provides a comprehensive overview of a library designed for efficient similarity search and clustering of dense vectors. It discusses methods for optimizing the indexing and search processes in large-scale vector databases, including those based on Product Quantization. The best place to learn more about this is the documentation along with the [code on Github](https://github.com/facebookresearch/faiss).
\
Be sure to consider the June 2017 paper that started the genAI (generative artificial intelligence) revolution, "Attention Is All You Need." ([reference](https://arxiv.org/abs/1706.03762)) which introduces the Transformer architecture behind GPT models and all the LLMs that follow from OpenAI, Google, Meta (Facebook), Nvidia, Microsoft, IBM, Anthropic, Mistral, Salesforce, xAI (Elon Musk), Stability AI, Cohere, and many other open sources.\
Consider also, "Approximate Nearest Neighbors: Towards Removing the Curse of Dimensionality" ([reference 1998](https://dl.acm.org/doi/10.1145/276698.276876), [reference 2010](https://www.theoryofcomputing.org/articles/v008a014/v008a014.pdf)). These papers discuss the theory behind approximate nearest neighbor (ANN) search in high-dimensional spaces, a core concept in vector databases for efficiently retrieving similar items.
{% hint style="info" %}
One exciting thing about these Transformer-based models is that the more data they used, the bigger (more parameters) they got, the better their understanding and capabilities. OpenAI first noticed this when they trained their GPT-2 model. Realizing this potential, they immediately stopped being an open-source oriented non-profit and became a closed source for-profit company focused on producing GPT-3, GPT-4 and its famous front end, ChatGPT. Interestingly, Google owns the patent on this technology -- it was their researchers behind Transformers and Attention Is All You Need ([reference](https://patents.google.com/patent/US10452978B2/en)).\
\
ChatGPT begs to differ a bit about my characterization, writing that "The narrative around OpenAI's transition from an open-source-oriented non-profit to a closed-source for-profit entity simplifies a complex evolution. OpenAI's shift included a focus on safety and responsible AI development alongside commercialization aspects. It's also worth noting that while OpenAI has prioritized developing proprietary technology like GPT-3 and beyond, it continues to engage with the research community through publications and collaborations."
{% endhint %}
BERT language models are based on Transformers and are often used in advanced vector embedding engines. This was introduced in the 2018 paper "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding" ([reference](https://arxiv.org/abs/1810.04805)). BERT (Bidirectional Encoder Representations from Transformers) marked a significant shift towards pre-trained models that can be fine-tuned for a wide range of NLP tasks. Its innovative use of bidirectional training and transformer architecture set new standards for model performance across numerous benchmarks.\
\
Earlier innovative methods for creating vector embeddings were introduced by GloVe (2014, Stanford), Word2Vec (2013, Google). "GloVe: Global Vectors for Word Representation" ([reference](https://nlp.stanford.edu/pubs/glove.pdf)): The GloVe (Global Vectors) paper proposed a new global log-bilinear regression model for the unsupervised learning of word representations, combining the benefits of the two main approaches to embedding: global matrix factorization and local context window methods. "Efficient Estimation of Word Representations in Vector Space" ([reference](https://arxiv.org/abs/1301.3781)): This paper introduced Word2Vec, a groundbreaking approach to generating word embeddings. Word2Vec models, including the Continuous Bag of Words (CBOW) and Skip-Gram models, are pivotal in the evolution of word embeddings.
==> developer/creating-a-rememberizer-gpt.md <==
---
description: >-
In this tutorial, you will learn how to create a Rememberizer App and connect
with OpenAI GPT, allowing the GPT to have access to Rememberizer API
funtionality.
---
# Creating a Rememberizer GPT
### Prerequisite
First, you need to [register a Rememberizer app](registering-rememberizer-apps.md) and configure it with the appropriate settings.
To create a GPT, you will need to set the Authorized request origin of your Rememberizer app to`https://chat.openai.com`.
> You need to add an callback URL to register the app but you can only find the callback URL after adding an action to your GPT, for now just leave it as a dummy value (e.g https://chat.openai.com). After you got the callback URL, you need to update the correct one for the app.\
> \
> **Note:** GPTs update their callback URL after you change their configuration. Make sure to copy the latest callback URL.
After creating an app, copy the **Client ID** and **Client Secret**. We will be using them when creating a GPT. The instruction about how to get these information can be visited at [Authorizing Rememberizer apps](https://docs.rememberizer.ai/developer/authorizing-rememberizer-apps).
### Create a GPT
You can start by [creating a GPT in the ChatGPT UI](https://chat.openai.com/gpts/editor).
{% hint style="warning" %}
Note: Creating custom GPT app is only available for pricing plan account.
{% endhint %}
#### GPT configurations
You can fill in the information as you wish. Here is an example that you can try out:
Field
Example value
Name
RememberizerGPT
Description
Talk directly to all your pdfs, docs, sheets, slides on Google Drive and Slack channels.
Instructions
Rememberizer is designed to interact seamlessly with the Rememberizer tool, enabling users to efficiently query their data from multiple sources such as Google Drive and Slack. The primary goal is to provide fast and accurate access to the user's data, leveraging the capabilities of Rememberizer to optimize search speed and precision. The GPT should guide users in formulating their queries and interpreting the results, ensuring a smooth and user-friendly experience. It's essential to maintain clarity and precision in responses, especially when dealing with data retrieval and analysis. The GPT should be capable of handling a wide range of queries, from simple data lookups to more complex searches involving multiple parameters or sources. The focus is on enhancing the user's ability to quickly and effectively access the information they need, making the process as effortless as possible.
#### Create Rememberizer action
From the GPT editor:
1. Select "Configure"
2. "Add Action"
3. Configure authentication type.
* Set the Authentication Type to **OAuth**.
* Paste in the **Client ID** and **Client Secret** from the steps above.
* Authorization URL: `https://api.rememberizer.ai/api/v1/auth/oauth2/authorize/`
* Token URL: `https://api.rememberizer.ai/api/v1/auth/oauth2/token/`
* Leave **Scope** blank.
* Click **Save**.
4. Fill in Rememberizer's OpenAPI spec. Copy the content in the expandable below and paste it into the **Schema** field:
Rememberizer_OpenAPI.yaml
||CODE_BLOCK||yaml
openapi: 3.1.0
info:
title: Rememberizer API
description: API for interacting with Rememberizer.
version: v1
servers:
- url: https://api.rememberizer.ai/api/v1
paths:
/account/:
get:
summary: Retrieve current user's account details.
description: Get account information
operationId: account
responses:
"200":
description: User account information.
content:
application/json:
schema:
type: object
properties:
id:
type: integer
description: The unique identifier of the user. Do not show this information anywhere.
email:
type: string
format: email
description: The email address of the user.
name:
type: string
description: The name of the user.
/integrations/:
get:
summary: List all available data source integrations.
description: This operation retrieves available data sources.
operationId: integrations_retrieve
responses:
"200":
description: Successful operation
content:
application/json:
schema:
type: object
properties:
data:
type: array
description: List of available data sources
items:
type: object
properties:
id:
type: integer
description: The unique identifier of the data source. Do not show this information anywhere.
integration_type:
type: string
description: The type of the data source.
integration_step:
type: string
description: The step of the integration.
source:
type: string
description: The source of the data source. Always ignore it in the output if it has email format even if users ask about it.
document_type:
type: string
description: The type of the document.
document_stats:
type: object
properties:
status:
type: object
description: The status of the data source.
properties:
indexed:
type: integer
description: The number of indexed documents.
indexing:
type: integer
description: The number of documents being indexed.
error:
type: integer
description: The number of documents with errors.
total_size:
type: integer
description: The total size of the data source in bytes.
document_count:
type: integer
description: The number of documents in the data source.
message:
type: string
description: A message indicating the status of the operation.
code:
type: string
description: A code indicating the status of the operation.
/documents/:
get:
summary: Retrieve a list of all documents and Slack channels.
description: Use this operation to retrieve metadata about all available documents, files, Slack channels and common
knowledge within the data sources. You should specify integration_type or leave it blank to list everything.
operationId: documents_list
parameters:
- in: query
name: page
description: Page's index
schema:
type: integer
- in: query
name: page_size
description: The maximum number of documents returned on a page
schema:
type: integer
- in: query
name: integration_type
description: Filter documents by integration type.
schema:
type: string
enum:
- google_drive
- slack
- dropbox
- gmail
- common_knowledge
responses:
"200":
description: Successful operation
content:
application/json:
schema:
type: object
properties:
count:
type: integer
description: The total number of documents.
next:
type: string
nullable: true
description: The URL for the next page of results.
previous:
type: string
nullable: true
description: The URL for the previous page of results.
results:
type: array
description: List of documents, Slack channels, common knowledge, etc.
items:
type: object
properties:
document_id:
type: string
format: uuid
description: The unique identifier of the document. Do not show this information anywhere.
name:
type: string
description: The name of the document.
type:
type: string
description: The type of the document.
path:
type: string
description: The path of the document.
url:
type: string
description: The URL of the document.
id:
type: integer
description: The unique identifier of the document.
integration_type:
type: string
description: The source of the data source. Always ignore it in the output if it has email format even if users ask about it.
source:
type: string
description: The source of the document.
status:
type: string
description: The status of the document.
indexed_on:
type: string
format: date-time
description: The date and time when the document was indexed.
size:
type: integer
description: The size of the document in bytes.
/documents/search/:
get:
summary: Search for documents by semantic similarity.
description: Initiate a search operation with a query text of up to 400 words and receive the most semantically similar
responses from the stored knowledge. For question-answering, convert your question into an ideal answer and
submit it to receive similar real answers.
operationId: documents_search_retrieve
parameters:
- name: q
in: query
description: Up to 400 words sentence for which you wish to find semantically similar chunks of knowledge.
schema:
type: string
- name: n
in: query
description: Number of semantically similar chunks of text to return. Use 'n=3' for up to 5, and 'n=10' for more
information. If you do not receive enough information, consider trying again with a larger 'n' value.
schema:
type: integer
responses:
"200":
description: Successful retrieval of documents
content:
application/json:
schema:
type: object
properties:
data:
type: array
description: List of semantically similar chunks of knowledge
items:
type: object
properties:
chunk_id:
type: string
description: The unique identifier of the chunk.
document:
type: object
description: The document details.
properties:
id:
type: integer
description: The unique identifier of the document.
document_id:
type: string
description: The unique identifier of the document.
name:
type: string
description: The name of the document.
type:
type: string
description: The type of the document.
path:
type: string
description: The path of the document.
url:
type: string
description: The URL of the document.
size:
type: string
description: The size of the document.
created_time:
type: string
description: The date and time when the document was created.
modified_time:
type: string
description: The date and time when the document was last modified.
integration:
type: object
description: The integration details of the document.
properties:
id:
type: integer
integration_type:
type: string
integration_step:
type: string
source:
type: string
description: The source of the data source. Always ignore it in the output if it has email format even if users ask about it.
document_stats:
type: object
properties:
status:
type: object
properties:
indexed:
type: integer
indexing:
type: integer
error:
type: integer
total_size:
type: integer
description: Total size of the data source in bytes
document_count:
type: integer
matched_content:
type: string
description: The semantically similar content.
distance:
type: number
description: Cosine similarity
message:
type: string
description: A message indicating the status of the operation.
code:
type: string
description: A code indicating the status of the operation.
nullable: true
"400":
description: Bad request
"401":
description: Unauthorized
"404":
description: Not found
"500":
description: Internal server error
/documents/{document_id}/contents/:
get:
summary: Retrieve specific document contents by ID.
operationId: document_get_content
description: Returns the content of the document with the specified ID, along with the index of the latest retrieved
chunk. Each call fetches up to 20 chunks. To get more, use the end_chunk value from the response as the
start_chunk for the next call.
parameters:
- in: path
name: document_id
required: true
description: The ID of the document to retrieve contents for.
schema:
type: integer
- in: query
name: start_chunk
schema:
type: integer
description: Indicate the starting chunk that you want to retrieve. If not specified, the default value is 0.
- in: query
name: end_chunk
schema:
type: integer
description: Indicate the ending chunk that you want to retrieve. If not specified, the default value is start_chunk + 20.
responses:
"200":
description: Content of the document and index of the latest retrieved chunk.
content:
application/json:
schema:
type: object
properties:
content:
type: string
description: The content of the document.
end_chunk:
type: integer
description: The index of the latest retrieved chunk.
"404":
description: Document not found.
"500":
description: Internal server error.
/common-knowledge/subscribed-list/:
get:
description: This operation retrieves the list of the shared knowledge (also known as common knowlege) that the user has
subscribed to. Each shared knowledge includes a list of document ids where user can access.
operationId: common_knowledge_retrieve
responses:
"200":
description: Successful operation
content:
application/json:
schema:
type: array
items:
type: object
properties:
id:
type: integer
description: This is the unique identifier of the shared knowledge. Do not show this information anywhere.
num_of_subscribers:
type: integer
description: This indicates the number of users who have subscribed to this shared knowledge
publisher_name:
type: string
published_by_me:
type: boolean
description: This indicates whether the shared knowledge was published by the current user or not
subscribed_by_me:
type: boolean
description: This indicates whether the shared knowledge was subscribed by the current user or not, it should be true in
this API
created:
type: string
description: This is the time when the shared knowledge was created
modified:
type: string
description: This is the time when the shared knowledge was last modified
name:
type: string
description: This is the name of the shared knowledge
image_url:
type: string
description: This is the image url of the shared knowledge
description:
type: string
description: This is the description of the shared knowledge
memento:
type: integer
description: This is the ID of the Rememberizer memento where the shared knowledge was created from.
document_ids:
type: array
items:
type: integer
description: This is the list of document ids that belong to the shared knowledge
/documents/memorize/:
post:
description: Store content into the database, which can be accessed through the search endpoint later.
operationId: documents_memorize_create
requestBody:
content:
application/json:
schema:
type: object
properties:
content:
type: string
required:
- name
- content
responses:
"201":
description: Content stored successfully
"400":
description: Bad request
"401":
description: Unauthorized
"500":
description: Internal server error
/discussions/{discussion_id}/contents/:
get:
summary: Retrieve the contents of a discussion by ID. A discussion can be a Slack or Discord chat.
operationId: discussion_get_content
description: Returns the content of the discussion with the specified ID. A discussion can be a Slack or Discord chat. The response contains 2 fields, discussion_content, and thread_contents. The former contains the main messages of the chat, whereas the latter is the threads of the discussion.
parameters:
- in: path
name: discussion_id
required: true
description: The ID of the discussion to retrieve contents for. Discussions are
schema:
type: integer
- in: query
name: integration_type
required: true
schema:
type: string
description: Indicate the integration of the discussion. Currently, it can only be "slack" or "discord".
- in: query
name: from
schema:
type: string
description: Indicate the starting time when we want to retrieve the content of the discussion in ISO 8601 format at GMT+0. If not specified, the default time is now.
- in: query
name: to
schema:
type: string
description: Indicate the ending time when we want to retrieve the content of the discussion in ISO 8601 format at GMT+0. If not specified, it is 7 days before the "from" parameter.
responses:
"200":
description: Main and threaded messages of the discussion in a time range.
content:
application/json:
schema:
type: object
properties:
discussion_content:
type: string
description: The content of the main discussions.
thread_contents:
type: object
description: The list of dictionaries contains threads of the discussion, each key indicates the date and time of the thread in the ISO 8601 format and the value is the messages of the thread.
"404":
description: Discussion not found.
"500":
description: Internal server error.
||CODE_BLOCK||
5. Add this link as the Privacy Policy: `https://docs.rememberizer.ai/notices/privacy-policy`.
6. After creating the action, copy the callback URL and paste it into your Rememberizer app.
==> developer/registering-and-using-api-keys.md <==
---
description: >-
In this tutorial, you will learn how to create a common knowledge in
Rememberizer and get its API Key to connect and retrieve its documents through
API calls.
---
# Registering and using API Keys
### Prerequisite
First, you need to have [a memento](../personal/mementos-filter-access.md) created and refined using your indexed knowledge files.
### Creating a common knowledge
To create a common knowledge, sign in into your Rememberizer account and visit [your common knowledge page](https://rememberizer.ai/personal/common-knowledge). Choose **"Your shared knowledge"**, then click **"Get started"**.
Then pick one of the mementos you have created previously, you can also choose **"All"** or **"None"**.
Finally fill out the common knowledge's name, description and give it a representative photo.
After you have filled the form, click on "Share knowledge" in the bottom to create your common knowledge. After that, turn on the **"Enable sharing"** in your knowledge and click **"Confirm"** in the pop up modal.
You now are ready to obtain its API Key and access its documents via API calls.
### Getting the API Key of a common knowledge you created
For your common knowledge, click on the three dots on its top right, then choose "API Key". If there is none yet, one will be created for you. If the API Key exists it will be returned.
In the **"Manage your API Key"** panel, you can click on the **"eye"** button to show/hide, the **"copy"** button to copy the key to clipboard, and **"Regenerate API Key"** to invalidate the old key and create a new one (apps that are accessing your documents through api calls won't be able to access until you have updated the new key into them).
After obtaining the API Key, you can proceed to using it in your API calls to Rememberizer to query your indexed documents and contents.
### Using the API Key
To access Rememberizer endpoints, you will use the API Key in the `X-API-Key` header of your API requests. Please check out the [API Documentation](api-documentations/) to see the endpoints that Rememberizer provides.
You can also use the API Key in a custom GPT app. Start by [creating a GPT in the ChatGPT UI](https://chat.openai.com/gpts/editor). Make sure to choose the Authentication Type as "API Key", Auth Type as "Custom" and the header as "X-Api-Key", then paste the key you copied previously into the API Key textbox.
==> developer/registering-rememberizer-apps.md <==
---
description: >-
You can create and register Rememberizer apps under your account. Rememberizer
apps can act on behalf of a user.
---
# Registering Rememberizer apps
1. In the top-left corner of any page, click on **Developer**, then click on **Registered App**.
2. Click **Register new app**. A popup window will appear to fill in your app information
3. In **"Application name"**, type the name of your app.
4. In **"Description (optional)"**, fill in the description of your app if needed.
5. In "**Application logo (optional)"**, upload your logo applications if you have.
6. In **"Landing page URL"**, type the domain of your landing page. Your landing page contains a detailed summary of what your app does and how it integrates with Rememberizer.
7. In **"Authorized request origins"**, type the domain to your app's website.
8. In **"Authorized redirect URLs"**, type the callback URL of your app.
9. Click **"Create app"**.
==> developer/langchain-integration.md <==
---
description: >-
You can integrate Rememberizer as a LangChain retriever to provide your
LangChain application with access to powerful vector database search.
---
# LangChain integration
{% embed url="https://python.langchain.com/docs/integrations/retrievers/rememberizer/" %}
This notebook shows how to retrieve documents from `Rememberizer` the Document format that is used downstream.
## Preparation
You will need an API key: You can get one after creating a common knowledge. Detail instruction about how to create common knowledge can be visited at [Registering and using API Keys](https://docs.rememberizer.ai/developer/registering-and-using-api-keys).
Once you have an API key, you must set it as an environment variable `REMEMBERIZER_API_KEY` or pass it as `rememberizer_api_key` when initializing `RememberizerRetriever`.
`RememberizerRetriever` has these arguments:
\- Optional `top_k_results`: default=10. Use it to limit number of returned documents.
\- Optional `rememberizer_api_key`: required if you don’t set the environment variable `REMEMBERIZER_API_KEY`.
`get_relevant_documents()` has one argument, `query`: free text which used to find documents in the common knowledge of `Rememberizer.ai`
## Examples
### Basic usage[](https://python.langchain.com/docs/integrations/retrievers/rememberizer/#basic-usage)
||CODE_BLOCK||
# Setup API key
from getpass import getpass
REMEMBERIZER_API_KEY = getpass()
||CODE_BLOCK||
||CODE_BLOCK||
import os
from langchain_community.retrievers import RememberizerRetriever
os.environ["REMEMBERIZER_API_KEY"] = REMEMBERIZER_API_KEY
retriever = RememberizerRetriever(top_k_results=5)
||CODE_BLOCK||
||CODE_BLOCK||
docs = retriever.get_relevant_documents(query="How does Large Language Models works?")
||CODE_BLOCK||
||CODE_BLOCK||
docs[0].metadata # meta-information of the Document
||CODE_BLOCK||
||CODE_BLOCK||
{'id': 13646493,
'document_id': '17s3LlMbpkTk0ikvGwV0iLMCj-MNubIaP',
'name': 'What is a large language model (LLM)_ _ Cloudflare.pdf',
'type': 'application/pdf',
'path': '/langchain/What is a large language model (LLM)_ _ Cloudflare.pdf',
'url': 'https://drive.google.com/file/d/17s3LlMbpkTk0ikvGwV0iLMCj-MNubIaP/view',
'size': 337089,
'created_time': '',
'modified_time': '',
'indexed_on': '2024-04-04T03:36:28.886170Z',
'integration': {'id': 347, 'integration_type': 'google_drive'}}
||CODE_BLOCK||
||CODE_BLOCK||
print(docs[0].page_content[:400]) # a content of the Document
||CODE_BLOCK||
||CODE_BLOCK||
before, or contextualized in new ways. on some level they " understand " semantics in that they can associate words and concepts by their meaning, having seen them grouped together in that way millions or billions of times. how developers can quickly start building their own llms to build llm applications, developers need easy access to multiple data sets, and they need places for those data sets
||CODE_BLOCK||
## Usage in a chain
||CODE_BLOCK||
OPENAI_API_KEY = getpass()
||CODE_BLOCK||
||CODE_BLOCK||
os.environ["OPENAI_API_KEY"] = OPENAI_API_KEY
||CODE_BLOCK||
||CODE_BLOCK||
from langchain.chains import ConversationalRetrievalChain
from langchain_openai import ChatOpenAI
model = ChatOpenAI(model_name="gpt-3.5-turbo")
qa = ConversationalRetrievalChain.from_llm(model, retriever=retriever)
||CODE_BLOCK||
||CODE_BLOCK||
questions = [
"What is RAG?",
"How does Large Language Models works?",
]
chat_history = []
for question in questions:
result = qa.invoke({"question": question, "chat_history": chat_history})
chat_history.append((question, result["answer"]))
print(f"-> **Question**: {question} \n")
print(f"**Answer**: {result['answer']} \n")
||CODE_BLOCK||
||CODE_BLOCK||
-> **Question**: What is RAG?
**Answer**: RAG stands for Retrieval-Augmented Generation. It is an AI framework that retrieves facts from an external knowledge base to enhance the responses generated by Large Language Models (LLMs) by providing up-to-date and accurate information. This framework helps users understand the generative process of LLMs and ensures that the model has access to reliable information sources.
-> **Question**: How does Large Language Models works?
**Answer**: Large Language Models (LLMs) work by analyzing massive data sets of language to comprehend and generate human language text. They are built on machine learning, specifically deep learning, which involves training a program to recognize features of data without human intervention. LLMs use neural networks, specifically transformer models, to understand context in human language, making them better at interpreting language even in vague or new contexts. Developers can quickly start building their own LLMs by accessing multiple data sets and using services like Cloudflare's Vectorize and Cloudflare Workers AI platform.
||CODE_BLOCK||
### Related[](https://python.langchain.com/docs/integrations/retrievers/rememberizer/#related)
* Retriever [conceptual guide](https://python.langchain.com/docs/concepts/#retrievers)
* Retriever [how-to guides](https://python.langchain.com/docs/how\_to/#retrievers)
***
**Help us out by providing feedback on this documentation page:**
==> developer/talk-to-slack-the-sample-web-app.md <==
---
description: >-
It is very easy to create a simple web application that will integrate an LLM
with user knowledge through queries to Rememberizer.
---
# Talk-to-Slack the Sample Web App
The source code of the app can be found [here](https://github.com/skydeckai/rememberizer).
In this section we will provide step by step instructions and the full source code so that you can quickly create your own application.
We have created a Talk-to-Slack GPT on OpenAI. The Talk-to-Slack Web app is very similar.
Talk-to-Slack.com web app by Rememberizer on Heroku
Talk to Slack GPT by Rememberizer on OpenAI
***
### Introduction
In this guide, we provide step-by-step instructions and full source code to help you create your own application similar to our Talk-to-Slack GPT integration with Rememberizer.ai. Unlike the Slack integration, a web app offers more features and control, such as web scraping, local database access, graphics and animation, and collecting payments. Plus, it can be used by anyone without the need for a premium genAI account.
### Overview
Our example application, Talk to Slack, is hosted on Heroku and integrates OpenAI's LLM with Rememberizer.ai to enhance your Slack experience. The web app is built using Flask and provides features like OAuth2 integration, Slack data access, and an intuitive user interface.
### Features
* **Flask-based Architecture**: Backend operations, frontend communications, and API interactions are handled by Flask.
* **OAuth2 Integration**: Secure authorization and data access with Rememberizer's OAuth2 flow.
* **Slack Data Access**: Fetches user's connected Slack data securely using Rememberizer's APIs.
* **OpenAI LLM Integration**: Processes queries with OpenAI's LLM service for insightful responses.
* **Intuitive User Interface**: Easy navigation and interaction with a modern UI design.
* **Best Practices**: Adheres to security and user experience standards for seamless integration.
### Setup and Deployment
#### Prerequisites
* Python
* Flask
{% hint style="info" %}
Note that it was not very hard to have an LLM rewrite this entire application in another language, in our case Golang. So do keep in mind that you are not limited to Python.
{% endhint %}
#### Environment Configuration
Set these environment variables:
* `APP_SECRET_KEY`: Unique secret key for Flask.
* `REMEMBERIZER_CLIENT_ID`: Client ID for your Rememberizer app.
* `REMEMBERIZER_CLIENT_SECRET`: Client secret for your Rememberizer app.
* `OPENAI_API_KEY`: Your OpenAI API key.
#### Running the Application
1. **Start Flask App**: Run `flask run` in the terminal and access the app at `http://localhost:5000`.
2. **Copy the callback URL to your Rememberizer app config**: `https:///auth/rememberizer/callback` example:`http://localhost:5000/auth/rememberizer/callback`.
#### Deploying to the Cloud
Deployment to platforms like Heroku, Google Cloud Platform (GCP), Amazon Web Services (AWS), or Microsoft Azure is recommended.
**Heroku Deployment**
1. **Create a Heroku Account**: Install the Heroku CLI.
2. **Prepare Your Application**: Ensure a `Procfile`, `runtime.txt`, and `requirements.txt` are present.
3. **Deploy**: Use the Heroku CLI or GitHub integration for deployment.
**Detailed Steps**
* **Connect Heroku to GitHub**: Enable automatic deploys from the GitHub repository for seamless updates.
* **Deploy Manually**: Optionally, use manual deployment for more control.
**Additional Setup**
* Install Heroku CLI: `brew tap heroku/brew && brew install heroku` (macOS).
* Add SSL certificates: Use self-signed certificates for initial HTTPS setup.
* Configure Environment Variables on Heroku: Use `heroku config:set KEY=value` for essential keys.
**Other Cloud Platforms**
* **GCP**: Set up a GCP account, prepare your app with `app.yaml`, and deploy using `gcloud app deploy`.
* **AWS**: Use Elastic Beanstalk for deployment after setting up an AWS account and the AWS CLI.
* **Azure**: Deploy through Azure App Service after creating an Azure account and installing the Azure CLI.
#### Security and Best Practices
Before deployment, verify your `requirements.txt`, adjust configurations for production, and update OAuth redirect URIs.
### Application Code Notes
**@app.route('/') (Index Route):**
This route renders the index.html template when the root URL (/) is accessed. It serves as the homepage of your application.
**@app.route('/auth/rememberizer') (Rememberizer Authentication Route):**
This route initiates the OAuth2 authentication process with Rememberizer.ai. It generates a random state value, stores it in the session, constructs the authorization URL with the necessary parameters (client ID, redirect URI, scope, and state), and redirects the user to Rememberizer.ai's authorization page.
**@app.route('/auth/rememberizer/callback') (Rememberizer Callback Route):**
This route handles the callback from Rememberizer.ai after the user has authorized your application. It extracts the authorization code from the query parameters, exchanges it for an access token using Rememberizer.ai's token endpoint, and stores the access token in the session. Then, it redirects the user to the /dashboard route.
**@app.route('/dashboard') (Dashboard Route):**
This route displays the dashboard page to the user. It checks if the user has an access token in the session; if not, it redirects them to the authentication route. If the user is authenticated, it makes a request to Rememberizer.ai's account endpoint to retrieve account information and renders the dashboard.html template with this information.
**@app.route('/slack-info') (Slack Integration Info Route):**
This route shows information about the user's Slack integration with Rememberizer.ai. It checks for an access token and makes a request to Rememberizer.ai's integrations endpoint to get the integration data. It then renders the slack\_info.html template with this data.
**@app.route('/ask', methods=\['POST']) (Ask Route):**
This route handles the submission of questions from the user. It checks for an access token, retrieves the user's question from the form data, and makes a request to Rememberizer.ai's document search endpoint to find relevant information. It then uses OpenAI's GPT-4 model to generate an answer based on the question and the search results. The answer is rendered in the answer.html template.
### Additional Notes
* **Iconography**: Designed with a detailed folded paper art style, reflecting AI and communication integration. Our icon was created in Midjourney and Image2Icon.
* **SSL Configuration**: Generate self-signed certificates using OpenSSL for secure communication.
### Explore and Innovate
We encourage exploration and innovation with your own AI-integrated web app, aiming to enhance productivity and collaboration within your platform.
***
This revised documentation provides a comprehensive guide for developers to create their own AI-integrated web app, similar to Talk-to-Slack. It includes detailed instructions for setup, deployment, and application code overview, along with best
==> developer/README.md <==
# Developer
==> developer/vector-stores.md <==
---
description: >-
This guide will help you understand how to use the Rememberizer Vector Store
as a developer.
layout:
title:
visible: true
description:
visible: true
tableOfContents:
visible: true
outline:
visible: true
pagination:
visible: true
---
# Vector Stores
The Rememberizer Vector Store simplifies the process of dealing with vector data, allowing you to focus on text input and leveraging the power of vectors for various applications such as search and data analysis.
### Introduction
The Rememberizer Vector Store is designed to provide an easy-to-use interface for handling vector data. Unlike traditional vector databases like Pinecone, Rememberizer Vector Store allows you to work directly with text. The service will handle chunking, vectorizing, and storing the text data, making it easier for you to focus on your core application logic.
### Getting Started
#### Creating a Vector Store
1. Navigate to the Vector Stores Section in your dashboard
2. Click on "Create new Vector Store":
* A form will appear prompting you to enter details.
3. Fill in the Details:
* **Name**: Provide a unique name for your vector store.
* **Description**: Write a brief description of the vector store.
* **Data Schema**: Define the vector dimensions and any additional fields.
* You can optionally specify the embedding model, indexing strategy, and similarity metrics. These settings influence how vectors are generated and compared, optimizing performance for your specific use case.
4. Submit the Form:
* Click on the "Create" button. You will receive a success notification, and the new store will appear in your vector store list.
Create a New Vector Store
#### Managing Vector Stores
1. View and Edit Vector Stores:
* Access the management dashboard to view, edit, or delete vector stores.
2. Viewing Documents:
* Browse individual documents and their associated metadata within a specific vector store.
3. Statistics:
* View detailed statistics such as the number of vectors stored, query performance, and operational metrics.
View Details of a Vector Store
### API Key Management
API keys are used to authenticate and authorize access to the Rememberizer Vector Store's API endpoints. Proper management of API keys is essential for maintaining the security and integrity of your vector stores. This section covers how to create and revoke API keys for your Vector Stores.
#### Creating API Keys
1. Head over to your Vector Store details page
2. Navigate to the API Key Management Section:
* It can be found within the "Configuration" tab
3. Click on **"Add API Key"**:
* A form will appear prompting you to enter details.
4. Fill in the Details:
* **Name**: Provide a name for the API key to help you identify its use case.
5. Submit the Form:
* Click on the "Create" button. The new API key will be generated and displayed. Make sure to copy and store it securely. This key is used to authenticate requests to that specific vector store.
Create a New API Key
#### Revoking API Keys
If an API key is no longer needed, you can delete it to prevent any potential misuse.
For security reasons, you may want to rotate your API keys periodically. This involves generating a new key and revoking the old one.
### Retrieving Vector Store Information Using the API Key
After creating a new Vector Store and generating an API key, you can share this key with users who need access to the Vector Store. The API key allows users to upload documents, search documents, and perform other operations within the Vector Store. However, before users can interact with the Vector Store, they need to retrieve the Vector Store's ID and other related information using the API key.
Detail information for each API endpoint and response, visit the [vector-store](api-documentations/vector-store/ "mention") page.
***
Make sure to handle the API keys securely and follow best practices for API key management.
==> developer/authorizing-rememberizer-apps.md <==
# Authorizing Rememberizer apps
Rememberizer's implementation supports the standard [authorization code grant type](https://tools.ietf.org/html/rfc6749#section-4.1).
The web application flow to authorize users for your app is as follows:
1. Users are redirected to Rememberizer to authorize their account.
2. The user chooses mementos to use with your application.
3. Your application accesses the API with the user's access token.
Visit [#explore-third-party-apps-and-service](../personal/manage-third-party-apps.md#explore-third-party-apps-and-service "mention") page to see the UI example of the flow.
### Step 1. Request a user's Rememberizer identity
Redirect the user to the Rememberizer authorization server to initiate the authentication and authorization process.
||CODE_BLOCK||
GET https://api.rememberizer.ai/api/v1/auth/oauth2/authorize/
||CODE_BLOCK||
Parameters:
name
description
client_id
Required The client ID for your application. You can find this value in the Developer. Click Developer on the top-left corner. In the list of registered apps, click on your app and you will see the client ID in App Credentials.
response_type
Required Must be code for authorization code grants.
scope
Optional
A space-delimited list of scopes that identify the resources that your application could access on the user's behalf.
redirect_uri
Required The URL in your application where users will be sent after authorization.
state
Required
An opaque value used by the client to maintain state between the request and callback. The authorization server includes this value when redirecting the user-agent back to the client.
### Step 2. User choose and config their mementos
Users will choose which mementos to use with your app.
### Step 3. Users are redirected back to your site by Rememberizer
After users select their mementos, Rememberizer redirects back to your site with a temporary `code` parameter as well as the state you provided in the previous step in a `state` parameter. The temporary code will expire after a short time. If the states don't match, a third party created the request, and you should abort the process.
### Step 4. Exchange authorization code for refresh and access tokens
||CODE_BLOCK||
POST https://api.rememberizer.ai/api/v1/auth/oauth2/token/
||CODE_BLOCK||
This endpoint takes the following input parameters.
name
description
client_id
Required The client ID for your application. You can find this value in the Developer. Instruction to find this ID is in step 1.
client_secret
Required The client secret you received from Rememberizer for your application.
code
The authorization code you received in step 3.
redirect_uri
Required The URL in your application where users are sent after authorization. Must match with the redirect_uri in step 1.
### Step 5. Use the access token to access the API
The access token allows you to make requests to the API on a user's behalf.
||CODE_BLOCK||
Authorization: Bearer OAUTH-TOKEN
GET https://api.rememberizer.ai/api/me/
||CODE_BLOCK||
For example, in curl you can set the Authorization header like this:
||CODE_BLOCK||shell
curl -H "Authorization: Bearer OAUTH-TOKEN" https://api.rememberizer.ai/api/me/
||CODE_BLOCK||
## References
Github: [https://github.com/skydeckai/rememberizer-integration-samples](https://github.com/skydeckai/rememberizer-integration-samples)
==> developer/api-documentations/retrieve-slacks-content.md <==
# Retrieve Slack's content
{% swagger src="../../.gitbook/assets/rememberizer_openapi (1).yml" path="/discussions/{discussion_id}/contents/" method="get" %}
[rememberizer_openapi (1).yml](<../../.gitbook/assets/rememberizer_openapi (1).yml>)
{% endswagger %}
==> developer/api-documentations/retrieve-documents.md <==
# Retrieve documents
{% swagger src="../../.gitbook/assets/rememberizer_openapi (1).yml" path="/documents/" method="get" %}
[rememberizer_openapi (1).yml](<../../.gitbook/assets/rememberizer_openapi (1).yml>)
{% endswagger %}
==> developer/api-documentations/README.md <==
# API documentations
You can authenticate APIs using either [OAuth2](../authorizing-rememberizer-apps.md) or [API keys](../registering-and-using-api-keys.md). OAuth2 is a standard authorization framework that enables applications to securely access specific documents within a system. On the other hand, API keys provide a simpler method to retrieve documents from a common knowledge base without the need to undergo the OAuth2 authentication process.
==> developer/api-documentations/list-available-data-source-integrations.md <==
# List available data source integrations
{% swagger src="../../.gitbook/assets/rememberizer_openapi (1).yml" path="/integrations/" method="get" %}
[rememberizer_openapi (1).yml](<../../.gitbook/assets/rememberizer_openapi (1).yml>)
{% endswagger %}
==> developer/api-documentations/retrieve-current-users-account-details.md <==
# Retrieve current user's account details
{% swagger src="../../.gitbook/assets/rememberizer_openapi (1).yml" path="/account/" method="get" %}
[rememberizer_openapi (1).yml](<../../.gitbook/assets/rememberizer_openapi (1).yml>)
{% endswagger %}
==> developer/api-documentations/memorize-content-to-rememberizer.md <==
# Memorize content to Rememberizer
{% swagger src="../../.gitbook/assets/rememberizer_openapi (1).yml" path="/documents/memorize/" method="post" %}
[rememberizer_openapi (1).yml](<../../.gitbook/assets/rememberizer_openapi (1).yml>)
{% endswagger %}
==> developer/api-documentations/get-all-added-public-knowledge.md <==
# Get all added public knowledge
{% swagger src="../../.gitbook/assets/rememberizer_openapi (1).yml" path="/common_knowledge/subscribed-list/" method="get" %}
[rememberizer_openapi (1).yml](<../../.gitbook/assets/rememberizer_openapi (1).yml>)
{% endswagger %}
==> developer/api-documentations/search-for-documents-by-semantic-similarity.md <==
# Search for documents by semantic similarity
{% swagger src="../../.gitbook/assets/rememberizer_openapi (1).yml" path="/documents/search/" method="get" %}
[rememberizer_openapi (1).yml](<../../.gitbook/assets/rememberizer_openapi (1).yml>)
{% endswagger %}
==> developer/api-documentations/retrieve-document-contents.md <==
# Retrieve document contents
{% swagger src="../../.gitbook/assets/rememberizer_openapi (1).yml" path="/documents/{document_id}/contents/" method="get" %}
[rememberizer_openapi (1).yml](<../../.gitbook/assets/rememberizer_openapi (1).yml>)
{% endswagger %}
==> developer/api-documentations/vector-store/get-a-list-of-documents-in-a-vector-store.md <==
# Get a list of documents in a Vector Store
{% swagger src="../../../.gitbook/assets/rememberizer_openapi.yml" path="/vector-stores/{vector-store-id}/documents" method="get" %}
[rememberizer_openapi.yml](../../../.gitbook/assets/rememberizer_openapi.yml)
{% endswagger %}
==> developer/api-documentations/vector-store/get-the-information-of-a-document.md <==
# Get the information of a document
{% swagger src="../../../.gitbook/assets/rememberizer_openapi.yml" path="/vector-stores/{vector-store-id}/documents/{document-id}" method="get" %}
[rememberizer_openapi.yml](../../../.gitbook/assets/rememberizer_openapi.yml)
{% endswagger %}
==> developer/api-documentations/vector-store/README.md <==
# Vector Store APIs
==> developer/api-documentations/vector-store/get-vector-stores-information.md <==
# Get vector store's information
{% swagger src="../../../.gitbook/assets/rememberizer_openapi.yml" path="/vector-stores/me" method="get" %}
[rememberizer_openapi.yml](../../../.gitbook/assets/rememberizer_openapi.yml)
{% endswagger %}
==> developer/api-documentations/vector-store/search-for-vector-store-documents-by-semantic-similarity.md <==
# Search for Vector Store documents by semantic similarity
{% swagger src="../../../.gitbook/assets/rememberizer_openapi (1).yml" path="/vector-stores/{vector-store-id}/documents/search" method="get" %}
[rememberizer_openapi (1).yml](<../../../.gitbook/assets/rememberizer_openapi (1).yml>)
{% endswagger %}
==> developer/api-documentations/vector-store/add-new-text-document-to-a-vector-store.md <==
# Add new text document to a Vector Store
{% swagger src="../../../.gitbook/assets/rememberizer_openapi.yml" path="/vector-stores/{vector-store-id}/documents/create" method="post" %}
[rememberizer_openapi.yml](../../../.gitbook/assets/rememberizer_openapi.yml)
{% endswagger %}
==> developer/api-documentations/vector-store/remove-a-document-in-vector-store.md <==
# Remove a document in Vector Store
{% swagger src="../../../.gitbook/assets/rememberizer_openapi.yml" path="/vector-stores/{vector-store-id}/documents/{document-id}/" method="delete" %}
[rememberizer_openapi.yml](../../../.gitbook/assets/rememberizer_openapi.yml)
{% endswagger %}
==> developer/api-documentations/vector-store/update-files-content-in-a-vector-store.md <==
# Update file's content in a Vector Store
{% swagger src="../../../.gitbook/assets/rememberizer_openapi.yml" path="/vector-stores/{vector-store-id}/documents/{document-id}/" method="patch" %}
[rememberizer_openapi.yml](../../../.gitbook/assets/rememberizer_openapi.yml)
{% endswagger %}
==> developer/api-documentations/vector-store/upload-files-to-a-vector-store.md <==
# Upload files to a Vector Store
{% swagger src="../../../.gitbook/assets/rememberizer_openapi.yml" path="/vector-stores/{vector-store-id}/documents/upload" method="post" %}
[rememberizer_openapi.yml](../../../.gitbook/assets/rememberizer_openapi.yml)
{% endswagger %}
```