simple-data-analysis
Fast DuckDB-powered TypeScript library for tabular analysis, geospatial analysis, network graph analysis, vector similarity search, AI, Google Sheets, and data
What is simple-data-analysis?
Fast DuckDB-powered TypeScript library for tabular analysis, geospatial analysis, network graph analysis, vector similarity search, AI, Google Sheets, and data visualization.
Overview of simple-data-analysis
What it is
Simple Data Analysis (SDA) is a TypeScript library for tabular, geospatial, vector, and AI-assisted data analysis. It runs on Node.js and is powered by DuckDB. The library is split into two packages: simple-data-analysis-core (core DuckDB functions) and simple-data-analysis (extends core with AI, Google Sheets, Google Cloud Storage, and charting features). It is maintained by Nael Shiab, a computational journalist and senior data producer for CBC News. The library is open source under the MIT license.
Key features
- DuckDB-powered: uses DuckDB for fast analytical queries, with lazy-loaded extensions for spatial, fuzzy matching, full-text search, and vector indexes.
- Chainable API: synchronous transformation methods queue operations; async observer methods (e.g.,
getData(),log(),writeData()) flush the queue in a single DuckDB statement. - Geospatial analysis: supports spatial joins, point creation, and GeoJSON I/O via the DuckDB spatial extension.
- AI integration: includes row-by-row AI processing, embeddings, vector similarity search, hybrid search, RAG, and natural language queries.
- External services: integrates with Google Sheets, Google Cloud Storage, and Datawrapper for charts and maps.
- Lazy dependencies: AI, Google, and charting dependencies load only when used, avoiding startup cost for core operations.
Use cases
- Tabular data analysis: loading, filtering, joining, summarizing, and exporting CSV/Parquet/JSON data.
- Geospatial analysis: processing point data, performing spatial joins, and exporting GeoJSON.
- Data journalism: end-to-end pipelines from data loading to visualization, all in TypeScript.
- AI-assisted analysis: using embeddings, vector search, and RAG for natural language queries over datasets.
- Integration with external services: reading/writing Google Sheets, uploading to Google Cloud Storage, and publishing to Datawrapper.
Getting started
Install the library via npm:bashnpm i @nshiab/simple-data-analysisTo start a new data project with essential folders and configurations:bashnpx @nshiab/setup-data-project
License and pricing
The library is open source under the MIT license. No pricing information is stated.
Based on the project's public README and repository metadata.