DanielSwift commited on
Commit
77f0110
·
1 Parent(s): a7585d3

Update README with 300M capability and clearer description

Browse files
Files changed (1) hide show
  1. README.md +27 -23
README.md CHANGED
@@ -4,48 +4,52 @@ emoji: ⚡
4
  colorFrom: blue
5
  colorTo: green
6
  sdk: gradio
7
- sdk_version: 4.39.0 # Or check your gradio version
8
  app_file: app.py
9
  pinned: false
10
  ---
11
 
12
- # FlexChunk: Enabling 100M×100M Out-of-Core SpMV
13
 
14
- This Hugging Face Space demonstrates **FlexChunk**, an algorithm for performing Sparse Matrix-Vector Multiplication (SpMV) on matrices potentially too large to fit in RAM.
15
 
16
- **Key Idea:** Divide the matrix into manageable horizontal chunks, process them sequentially, and use minimal memory (~1.7 GB for 100M×100M SpMV).
17
 
18
  ## Interactive Demo
19
 
20
- The app above allows you to:
21
- 1. Generate a sparse matrix (adjust **Size**, **Density**, **Chunks**, **Challenging** flag).
22
- 2. Run SpMV using **FlexChunk** (chunking to disk) and **SciPy** (emulating disk load).
23
- 3. Compare performance (timings) and correctness.
24
 
25
- **Note:** Larger matrices/densities will take longer. Limits are set for public infrastructure.
 
 
 
 
26
 
27
  ## Performance Highlights
28
 
29
- FlexChunk demonstrates near-linear scaling in time and memory for increasing matrix dimensions.
30
-
31
- **Time Performance and Memory Usage:** See the original article for detailed performance graphs and analysis.
32
-
33
- ### Benchmark Results
34
-
35
- Selected results showing performance on large matrices:
36
 
37
  | Matrix Size | Non-zero Elements | Total Time | Peak RAM Usage |
38
  |--------------------|-------------------|-----------------|----------------|
39
- | ... | ... | ... | ... |
40
  | 1.0M × 1.0M | 1.2M | 1.07 s | 17.00 MB |
41
  | 10.0M × 10.0M | 12.0M | 10.21 s | 170.00 MB |
42
- | 30.0M × 30.0M | 36.0M | 31.13 s | 510.00 MB |
43
  | 50.0M × 50.0M | 62.5M | 55.27 s | 850.00 MB |
44
- | 70.0M × 70.0M | 88.2M | 1 min 17.1 s | 1.19 GB |
45
- | **100.0M × 100.0M**| **120.0M** | **1 min 47.1 s**| **1.70 GB** |
 
 
 
 
 
 
 
 
46
 
47
- *(Full table in the original [README](https://github.com/your-repo/FlexChunk/blob/main/README.md) - **Update this link!**)*
48
 
49
- ## More Information
50
 
51
- See the original post for technical details: [FlexChunk: Enabling 100M×100M Out-of-Core SpMV](https://www.lesswrong.com/posts/zpRhsdDkWygTDScxb/flexchunk-enabling-100m-100m-out-of-core-spmv-1-8-min-1-7-gb)
 
 
4
  colorFrom: blue
5
  colorTo: green
6
  sdk: gradio
7
+ sdk_version: 4.39.0
8
  app_file: app.py
9
  pinned: false
10
  ---
11
 
12
+ # FlexChunk: Enabling Massive Out-of-Core SpMV
13
 
14
+ This interactive demo showcases **FlexChunk**, an algorithm for performing Sparse Matrix-Vector Multiplication (SpMV) on matrices too large to fit in RAM.
15
 
16
+ **Key Benefit:** Process matrices up to 300M×300M using only ~1.7 GB RAM by dividing them into manageable horizontal chunks.
17
 
18
  ## Interactive Demo
19
 
20
+ The application above provides two modes:
21
+ - **Standard Mode**: For matrices up to 200K×200K with optional SciPy comparison
22
+ - **Advanced Mode**: For matrices up to 300M×300M using FlexChunk only
 
23
 
24
+ You can adjust parameters including:
25
+ - Matrix size
26
+ - Density
27
+ - Number of chunks
28
+ - Matrix type (standard or challenging with extreme values)
29
 
30
  ## Performance Highlights
31
 
32
+ FlexChunk demonstrates near-linear scaling in both time and memory usage:
 
 
 
 
 
 
33
 
34
  | Matrix Size | Non-zero Elements | Total Time | Peak RAM Usage |
35
  |--------------------|-------------------|-----------------|----------------|
 
36
  | 1.0M × 1.0M | 1.2M | 1.07 s | 17.00 MB |
37
  | 10.0M × 10.0M | 12.0M | 10.21 s | 170.00 MB |
 
38
  | 50.0M × 50.0M | 62.5M | 55.27 s | 850.00 MB |
39
+ | 100.0M × 100.0M | 120.0M | 1 min 47.1 s | 1.70 GB |
40
+
41
+ Our algorithm scales linearly to even larger matrices (up to 300M×300M) with proportional increases in processing time and memory.
42
+
43
+ ## How It Works
44
+
45
+ FlexChunk operates in three main stages:
46
+ 1. **Matrix Division**: Splits the matrix into horizontal chunks and saves to disk
47
+ 2. **Sequential Processing**: Loads one chunk at a time to minimize memory usage
48
+ 3. **Result Accumulation**: Combines partial results into the final vector
49
 
50
+ This approach makes it possible to multiply vectors with matrices that would otherwise exceed available RAM.
51
 
52
+ ## Links
53
 
54
+ - **Source Code**: [GitHub Repository](https://github.com/DanielSwift1992/FlexChunk)
55
+ - **Full Article**: [FlexChunk: Enabling 100M×100M Out-of-Core SpMV](https://www.lesswrong.com/posts/zpRhsdDkWygTDScxb/flexchunk-enabling-100m-100m-out-of-core-spmv-1-8-min-1-7-gb)