RAG - Creating our very first embedding

scrimba

Watch in fullscreen

by Per Harald Borgen

Intro to Mistral AI

About

Comments

Notes

RAG - Challenge: embedding all chunks and preparing it for the vector db4:40

RAG - Creating our very first embedding

by Per Harald Borgen

Intro to Mistral AI

Expand for more info

RAG - Challenge: embedding all chunks and preparing it for the vector db4:40

index.js

run

preview

console

import { RecursiveCharacterTextSplitter } from "langchain/text_splitter";
import MistralClient from "@mistralai/mistralai";
const client = new MistralClient(process.env.MISTRAL_API_KEY);

async function splitDocument(path) {
    const response = await fetch(path);
    const text = await response.text();
    const splitter = new RecursiveCharacterTextSplitter({
        chunkSize: 250,
        chunkOverlap: 40
    });
    const output = await splitter.createDocuments([text]);
    const textArr = output.map(chunk => chunk.pageContent);
    return textArr;
}

console.log(await splitDocument('handbook.txt'));

const client = new MistralClient(process.env.MISTRAL_API_KEY);

async function splitDocument(path) {

const response = await fetch(path);

const text = await response.text();

const splitter = new RecursiveCharacterTextSplitter({

chunkSize: 250,

chunkOverlap: 40

});

const output = await splitter.createDocuments([text]);

const textArr = output.map(chunk => chunk.pageContent);

return textArr;

}

console.log(await splitDocument('handbook.txt'));

import { RecursiveCharacterTextSplitter } from "langchain/text_splitter";

import MistralClient from "@mistralai/mistralai";

›

[

"I. MISSION II. OVERVIEW The Ambrics C-Corp Employee Handbook (the “Handbook”) has been developed to provide general guidelines about Ambrics policies and procedures for employees. It is a guide to assist you in becoming familiar with some of the"

"privileges and obligations of your employment, including Ambricsʹs policy of voluntary at‐will employment. None of the policies or guidelines in the Handbook are"

"intended to give rise to contractual rights or obligations, or to be construed as a guarantee of employment for any specific period of time, or any specific type of work. Additionally, with the"

"exception of the voluntary at‐will employment policy, these guidelines are subject to modification, amendment or revocation by Ambrics at any time, without advance notice. The personnel polices of Ambrics are established by the Board of"

"Directors, which has delegated authority and responsibility for their administration to the Executive Director. The Executive Director may, in turn, delegate authority for administering"

"specific policies. Employees are encouraged to consult the Executive Director for additional information regarding the policies, procedures, and privileges described in this Handbook."

"Questions about personnel matters also may be reviewed with the Executive Director. Ambrics will provide each individual a copy of this Handbook upon employment. All employees are expected to abide by it. The highest standards of personal and"

"professional ethics and behavior are expected of all Ambrics employees. Further, Ambrics expects each employee to display good judgment,"

"diplomacy and courtesy in their professional relationships with members of Ambricsʹs Board of Directors, committees, membership, staff, and the general public. 2 III. VOLUNTARY AT‐WILL EMPLOYMENT"

"2 III. VOLUNTARY AT‐WILL EMPLOYMENT Unless an employee has a written employment agreement with Ambrics, which provides differently, all employment at Ambrics is “at‐will.”"

"That means that employees may be terminated from employment with Ambrics with or without cause, and employees are free to leave the employment of Ambrics with or without cause. Any representation by any"

"Ambrics officer or employee contrary to this policy is not binding upon Ambrics unless it is in writing and is signed by the Executive Director with the approval of the Board of Directors. IV. EQUAL EMPLOYMENT OPPORTUNITY"

"IV. EQUAL EMPLOYMENT OPPORTUNITY Ambrics shall follow the spirit and intent of all federal, state and local employment law and is committed to equal employment opportunity. To that end,"

"the Board of Directors and Executive Director of Ambrics will not discriminate against any employee or applicant in a manner that violates the law. Ambrics is committed to providing equal opportunity for all employees and"

"applicants without regard to race, color, religion, national origin, sex, age, marital status, sexual orientation, disability, political affiliation, personal appearance, family responsibilities,"

"matriculation or any other characteristic protected under federal, state or local law. Each person is evaluated on the basis of personal skill and merit. Ambrics’s"

"policy regarding equal employment opportunity applies to all aspects of employment, including recruitment, hiring, job assignments, promotions, working conditions, scheduling,"

"benefits, wage and salary administration, disciplinary action, termination, and social, educational and recreational programs. The Executive Director shall act as the responsible"

"agent in the full implementation of the Equal Employment Opportunity policy. Ambrics will not tolerate any form of unlawful discrimination. All employees are expected to cooperate fully in implementing this policy. In"

"particular, any employee who believes that any other employee of Ambrics may have violated the Equal Employment Opportunity Policy should report the possible violation to the Executive Director."

"violation to the Executive Director. If Ambrics determines that a violation of this policy has occurred, it will take appropriate disciplinary action against the offending party, which can include"

"counseling, warnings, suspensions, and termination. Employees who report, in good faith, violations of this policy and employees who cooperate with investigations into alleged"

"violations of this policy will not be subject to retaliation. Upon completion of the investigation, Ambrics will inform the employee who made the complaint of the results of the investigation. 3"

"the investigation. 3 Ambrics is also committed to complying fully with applicable disability discrimination laws, and ensuring that equal opportunity in employment exists at Ambrics for qualified persons with disabilities. All employment practices"

"and activities are conducted on a non‐discriminatory basis. Reasonable accommodations will be available to all qualified disabled employees, upon request, so long as the potential"

"accommodation does not create an undue hardship on Ambrics. Employees who believe that they may require an accommodation should discuss these needs with the Executive Director."

"Executive Director. If you have any questions regarding this policy, please contact the Executive Director. V. POLICY AGAINST WORKPLACE HARASSMENT Ambrics is committed to providing a work environment for all"

"employees that is free from sexual harassment and other types of discriminatory harassment. Employees are expected to conduct themselves in a professional manner and to show respect for their co‐workers."

"for their co‐workers. Ambrics’s commitment begins with the recognition and acknowledgment that sexual harassment and other types of discriminatory harassment are, of course, unlawful. To reinforce this commitment, Ambrics has developed a"

"policy against harassment and a reporting procedure for employees who have been subjected to or witnessed harassment. This policy applies to all work‐related settings and activities, whether"

, ...]

-2:28