Spaces:
Sleeping
Sleeping
Commit
·
77f0110
1
Parent(s):
a7585d3
Update README with 300M capability and clearer description
Browse files
README.md
CHANGED
@@ -4,48 +4,52 @@ emoji: ⚡
|
|
4 |
colorFrom: blue
|
5 |
colorTo: green
|
6 |
sdk: gradio
|
7 |
-
sdk_version: 4.39.0
|
8 |
app_file: app.py
|
9 |
pinned: false
|
10 |
---
|
11 |
|
12 |
-
# FlexChunk: Enabling
|
13 |
|
14 |
-
This
|
15 |
|
16 |
-
**Key
|
17 |
|
18 |
## Interactive Demo
|
19 |
|
20 |
-
The
|
21 |
-
|
22 |
-
|
23 |
-
3. Compare performance (timings) and correctness.
|
24 |
|
25 |
-
|
|
|
|
|
|
|
|
|
26 |
|
27 |
## Performance Highlights
|
28 |
|
29 |
-
FlexChunk demonstrates near-linear scaling in time and memory
|
30 |
-
|
31 |
-
**Time Performance and Memory Usage:** See the original article for detailed performance graphs and analysis.
|
32 |
-
|
33 |
-
### Benchmark Results
|
34 |
-
|
35 |
-
Selected results showing performance on large matrices:
|
36 |
|
37 |
| Matrix Size | Non-zero Elements | Total Time | Peak RAM Usage |
|
38 |
|--------------------|-------------------|-----------------|----------------|
|
39 |
-
| ... | ... | ... | ... |
|
40 |
| 1.0M × 1.0M | 1.2M | 1.07 s | 17.00 MB |
|
41 |
| 10.0M × 10.0M | 12.0M | 10.21 s | 170.00 MB |
|
42 |
-
| 30.0M × 30.0M | 36.0M | 31.13 s | 510.00 MB |
|
43 |
| 50.0M × 50.0M | 62.5M | 55.27 s | 850.00 MB |
|
44 |
-
|
|
45 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
46 |
|
47 |
-
|
48 |
|
49 |
-
##
|
50 |
|
51 |
-
|
|
|
|
4 |
colorFrom: blue
|
5 |
colorTo: green
|
6 |
sdk: gradio
|
7 |
+
sdk_version: 4.39.0
|
8 |
app_file: app.py
|
9 |
pinned: false
|
10 |
---
|
11 |
|
12 |
+
# FlexChunk: Enabling Massive Out-of-Core SpMV
|
13 |
|
14 |
+
This interactive demo showcases **FlexChunk**, an algorithm for performing Sparse Matrix-Vector Multiplication (SpMV) on matrices too large to fit in RAM.
|
15 |
|
16 |
+
**Key Benefit:** Process matrices up to 300M×300M using only ~1.7 GB RAM by dividing them into manageable horizontal chunks.
|
17 |
|
18 |
## Interactive Demo
|
19 |
|
20 |
+
The application above provides two modes:
|
21 |
+
- **Standard Mode**: For matrices up to 200K×200K with optional SciPy comparison
|
22 |
+
- **Advanced Mode**: For matrices up to 300M×300M using FlexChunk only
|
|
|
23 |
|
24 |
+
You can adjust parameters including:
|
25 |
+
- Matrix size
|
26 |
+
- Density
|
27 |
+
- Number of chunks
|
28 |
+
- Matrix type (standard or challenging with extreme values)
|
29 |
|
30 |
## Performance Highlights
|
31 |
|
32 |
+
FlexChunk demonstrates near-linear scaling in both time and memory usage:
|
|
|
|
|
|
|
|
|
|
|
|
|
33 |
|
34 |
| Matrix Size | Non-zero Elements | Total Time | Peak RAM Usage |
|
35 |
|--------------------|-------------------|-----------------|----------------|
|
|
|
36 |
| 1.0M × 1.0M | 1.2M | 1.07 s | 17.00 MB |
|
37 |
| 10.0M × 10.0M | 12.0M | 10.21 s | 170.00 MB |
|
|
|
38 |
| 50.0M × 50.0M | 62.5M | 55.27 s | 850.00 MB |
|
39 |
+
| 100.0M × 100.0M | 120.0M | 1 min 47.1 s | 1.70 GB |
|
40 |
+
|
41 |
+
Our algorithm scales linearly to even larger matrices (up to 300M×300M) with proportional increases in processing time and memory.
|
42 |
+
|
43 |
+
## How It Works
|
44 |
+
|
45 |
+
FlexChunk operates in three main stages:
|
46 |
+
1. **Matrix Division**: Splits the matrix into horizontal chunks and saves to disk
|
47 |
+
2. **Sequential Processing**: Loads one chunk at a time to minimize memory usage
|
48 |
+
3. **Result Accumulation**: Combines partial results into the final vector
|
49 |
|
50 |
+
This approach makes it possible to multiply vectors with matrices that would otherwise exceed available RAM.
|
51 |
|
52 |
+
## Links
|
53 |
|
54 |
+
- **Source Code**: [GitHub Repository](https://github.com/DanielSwift1992/FlexChunk)
|
55 |
+
- **Full Article**: [FlexChunk: Enabling 100M×100M Out-of-Core SpMV](https://www.lesswrong.com/posts/zpRhsdDkWygTDScxb/flexchunk-enabling-100m-100m-out-of-core-spmv-1-8-min-1-7-gb)
|