feat: add guide for later usecases
Browse files- guide.md +80 -0
- metadata.jsonl +0 -0
- test.ipynb +0 -0
guide.md
ADDED
@@ -0,0 +1,80 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
# Gaia Agent Evaluation Guide
|
2 |
+
|
3 |
+
This guide will walk you through the setup process for running the sample code and evaluating your agent using Gaia results.
|
4 |
+
|
5 |
+
## Step 1: Configure API Keys
|
6 |
+
|
7 |
+
Before anything else, make sure you configure your secret keys in the **Space Settings** section.
|
8 |
+
|
9 |
+
- Log into each required platform.
|
10 |
+
- Locate and input your API keys in the designated fields.
|
11 |
+
|
12 |
+
## Step 2: Set Up Supabase
|
13 |
+
|
14 |
+
1. **Log in to Supabase**.
|
15 |
+
2. Navigate to your **space**, then go to your **project**.
|
16 |
+
3. Open the **SQL Editor**, paste the SQL code below, and run it to create the necessary table and function.
|
17 |
+
|
18 |
+
### 📦 SQL Code – Creating Tables and Functions
|
19 |
+
|
20 |
+
```sql
|
21 |
+
-- Enable pgvector if not already enabled
|
22 |
+
create extension if not exists vector;
|
23 |
+
|
24 |
+
-- Create the documents table (if not already done)
|
25 |
+
create table if not exists documents (
|
26 |
+
id bigserial primary key,
|
27 |
+
content text,
|
28 |
+
metadata jsonb,
|
29 |
+
embedding vector(768) -- Make sure this matches your model's embedding dimension
|
30 |
+
);
|
31 |
+
|
32 |
+
-- Create the match_documents function
|
33 |
+
create or replace function match_documents (
|
34 |
+
query_embedding vector(768),
|
35 |
+
match_count int default 5,
|
36 |
+
filter jsonb default '{}'
|
37 |
+
)
|
38 |
+
returns table (
|
39 |
+
id bigint,
|
40 |
+
content text,
|
41 |
+
metadata jsonb,
|
42 |
+
similarity float
|
43 |
+
)
|
44 |
+
language plpgsql
|
45 |
+
as $$
|
46 |
+
begin
|
47 |
+
return query
|
48 |
+
select
|
49 |
+
id,
|
50 |
+
content,
|
51 |
+
metadata,
|
52 |
+
1 - (embedding <=> query_embedding) as similarity
|
53 |
+
from documents
|
54 |
+
where metadata @> filter
|
55 |
+
order by embedding <=> query_embedding
|
56 |
+
limit match_count;
|
57 |
+
end;
|
58 |
+
$$;
|
59 |
+
```
|
60 |
+
4. After running the above, execute this command to ensure Supabase’s API layer (PostgREST) refreshes its internal schema cache:
|
61 |
+
```sql
|
62 |
+
NOTIFY pgrst, 'reload config';
|
63 |
+
```
|
64 |
+
## Step 3: Populate the Database
|
65 |
+
|
66 |
+
To enable document retrieval, you need to populate the database with example entries:
|
67 |
+
|
68 |
+
- Open and run the **test.ipynb** Jupyter notebook.
|
69 |
+
|
70 |
+
- This script reads from the **metadata.jsonl** file and inserts the examples into the documents table.
|
71 |
+
|
72 |
+
- This adds a Basic Retrieval capability to your agent, enhancing its performance.
|
73 |
+
|
74 |
+
## Step 4: Run the Evaluation
|
75 |
+
|
76 |
+
Once the database is set up and filled with data:
|
77 |
+
|
78 |
+
- Proceed to the Evaluation section in your project.
|
79 |
+
|
80 |
+
- Run the evaluation script to test and score your agent’s performance.
|
metadata.jsonl
ADDED
The diff for this file is too large to render.
See raw diff
|
|
test.ipynb
ADDED
The diff for this file is too large to render.
See raw diff
|
|