diff --git a/CHANGELOG.md b/CHANGELOG.md --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -0,0 +1,17 @@ +# eventium-sqlite Changelog + +## 0.2.1 (Unreleased) + +- `sqliteCheckpointStore` -- SQLite-backed `CheckpointStore` for `SequenceNumber` tracking. +- Re-exports `CheckpointName` from `eventium-sql-common`. + +## 0.2.0 (Unreleased) + +- Updated for eventium-core 0.2.0 API changes. +- Single `events` table with metadata columns (event_type, correlation_id, causation_id, created_at). +- `initializeSqliteEventStore` creates UUID index on the events table. +- Enabled `NoFieldSelectors`, `DuplicateRecordFields`, `OverloadedRecordDot` default extensions. + +## 0.1.0 + +Initial release. diff --git a/README.md b/README.md --- a/README.md +++ b/README.md @@ -1,222 +1,78 @@ # Eventium SQLite -SQLite-based event store implementation for embedded and single-process applications. +SQLite event store backend for Eventium. ## Overview -`eventium-sqlite` provides a lightweight, file-based event store implementation using SQLite. It's perfect for single-process applications, embedded systems, mobile apps, and scenarios where you want persistent storage without the complexity of a database server. - -## Features - -- ✅ **Zero Configuration** - No database server required -- ✅ **File-Based Storage** - Single database file for easy backup -- ✅ **ACID Transactions** - Full consistency guarantees from SQLite -- ✅ **Optimistic Concurrency** - Version-based conflict detection -- ✅ **Type-Safe Access** - Uses Persistent library -- ✅ **Cross-Platform** - Works on Linux, macOS, Windows -- ✅ **Embedded-Friendly** - Low resource footprint -- ✅ **Easy Deployment** - No separate database process - -## When to Use SQLite - -### ✅ Good Fit -- **Desktop Applications** - Local data storage -- **CLI Tools** - Persistent command-line applications -- **Mobile Apps** - Embedded event storage -- **Development/Testing** - Persistent data without server setup -- **Single-Process Systems** - No concurrent process access needed -- **Edge Computing** - Resource-constrained environments - -### ⚠️ Consider Alternatives -- **Multi-Process Systems** - Use `eventium-postgresql` instead -- **High Write Concurrency** - PostgreSQL handles concurrent writes better -- **Distributed Systems** - Need a client-server database -- **Very Large Datasets** - PostgreSQL scales better for TB+ data - -## Installation - -Add to your `package.yaml`: - -```yaml -dependencies: - - eventium-core - - eventium-sql-common - - eventium-sqlite - - persistent-sqlite # SQLite driver -``` - -## Usage - -```haskell -import Eventium.Store.Sqlite -import Database.Persist.Sqlite - -main :: IO () -main = do - -- Use file-based storage - withSqlitePool "events.db" 1 $ \pool -> do - -- Initialize schema - flip runSqlPool pool $ do - runMigration migrateAll - - -- Create event store - let store = makeSqliteEventStore pool - - -- Use with command handlers - result <- applyCommandHandler - (eventStoreWriter store) - (eventStoreReader store) - commandHandler - aggregateId - command -``` - -## Database Location - -### File-Based Storage -```haskell --- Relative path -withSqlitePool "events.db" 1 $ \pool -> ... - --- Absolute path -withSqlitePool "/var/lib/myapp/events.db" 1 $ \pool -> ... - --- User-specific location -home <- getHomeDirectory -let dbPath = home ".myapp" "events.db" -withSqlitePool dbPath 1 $ \pool -> ... -``` - -### In-Memory Storage (Testing) -```haskell --- Temporary in-memory database -withSqlitePool ":memory:" 1 $ \pool -> ... -``` - -## Configuration - -### Connection Pool -SQLite works best with a single connection per process: - -```haskell --- Recommended for SQLite -withSqlitePool "events.db" 1 $ \pool -> ... -``` +`eventium-sqlite` provides a lightweight, file-based event store using SQLite. +Ideal for CLI tools, single-process applications, and development environments +where a full database server is unnecessary. -### WAL Mode (Recommended) -Enable Write-Ahead Logging for better concurrent read performance: +## API ```haskell -withSqlitePool "events.db" 1 $ \pool -> do - flip runSqlPool pool $ do - rawExecute "PRAGMA journal_mode=WAL;" [] - runMigration migrateAll -``` - -Benefits: -- Readers don't block writers -- Better performance for read-heavy workloads -- Safer concurrent access - -## Performance - -Typical SQLite event store performance: -- **Writes**: ~1000-3000 events/sec -- **Reads**: ~5000-20000 events/sec -- **Storage**: ~1KB per event (JSON serialized) - -### Optimization Tips - -1. **Use WAL Mode** - Better concurrent access -2. **Batch Writes** - Multiple events per transaction -3. **Index Strategy** - Default indexes cover common queries -4. **VACUUM Regularly** - Reclaim space from deleted data -5. **Synchronous Mode** - Balance durability vs speed - -## Backup & Recovery - -### Simple File Copy -```bash -# Stop application or ensure no writes -cp events.db events.db.backup - -# Or use SQLite backup command -sqlite3 events.db ".backup events.db.backup" -``` +sqliteEventStoreWriter + :: (MonadIO m) + => SqlEventStoreConfig entity serialized + -> VersionedEventStoreWriter (SqlPersistT m) serialized -### Continuous Backup -```bash -# With WAL mode, backup while app runs -sqlite3 events.db ".backup events.db.backup" +initializeSqliteEventStore + :: (MonadIO m) + => SqlEventStoreConfig entity serialized + -> ConnectionPool + -> m () ``` -## Migration from In-Memory +Readers come from `eventium-sql-common` (re-exported): ```haskell --- Development: in-memory -development :: IO () -development = withSqlitePool ":memory:" 1 $ \pool -> ... - --- Production: file-based -production :: IO () -production = withSqlitePool "events.db" 1 $ \pool -> ... +sqlEventStoreReader :: SqlEventStoreConfig entity serialized -> VersionedEventStoreReader (SqlPersistT m) serialized +sqlGlobalEventStoreReader :: SqlEventStoreConfig entity serialized -> GlobalEventStoreReader (SqlPersistT m) serialized ``` -## Example: Complete CLI Application +## Usage ```haskell import Eventium.Store.Sqlite -import System.Directory (getAppUserDataDirectory) +import Database.Persist.Sqlite main :: IO () -main = do - -- Store in application data directory - dataDir <- getAppUserDataDirectory "myapp" - createDirectoryIfMissing True dataDir - - let dbPath = dataDir "events.db" - - withSqlitePool dbPath 1 $ \pool -> do - -- Initialize on first run - flip runSqlPool pool $ do - rawExecute "PRAGMA journal_mode=WAL;" [] - runMigration migrateAll - - -- Run application - runApp pool +main = runNoLoggingT $ do + pool <- createSqlitePool "events.db" 1 + liftIO $ initializeSqliteEventStore defaultSqlEventStoreConfig pool + + let writer = sqliteEventStoreWriter defaultSqlEventStoreConfig + reader = sqlEventStoreReader defaultSqlEventStoreConfig + -- writer and reader operate in SqlPersistT m ``` -## Tools +`initializeSqliteEventStore` runs migrations and creates a UUID index. -### SQLite CLI -```bash -# Open database -sqlite3 events.db +## When to Use -# Inspect schema -.schema +| Scenario | Recommendation | +|----------|----------------| +| CLI tools, desktop apps | SQLite | +| Development / prototyping | SQLite or Memory | +| Multi-process / production | PostgreSQL | +| Unit tests (no persistence needed) | Memory | -# Query events -SELECT * FROM events ORDER BY version DESC LIMIT 10; +## Installation -# Check database size -.dbinfo +```yaml +dependencies: + - eventium-core + - eventium-sqlite + - persistent-sqlite ``` -## Limitations - -- **Single Writer** - Only one process should write at a time -- **File Locking** - May have issues on network filesystems -- **Database Size** - Practical limit around 100GB-1TB -- **Concurrent Writes** - Limited compared to PostgreSQL - ## Documentation -- [Main README](../README.md) - Project overview -- [SQL Common](../eventium-sql-common/) - Shared SQL utilities -- [Design Documentation](../DESIGN.md) - Architecture details -- [Examples](../examples/) - Bank and cafe examples use SQLite +- [Main README](../README.md) +- [SQL Common](../eventium-sql-common/) +- [Design](../DESIGN.md) +- [Cafe Example](../examples/cafe/) and [Bank Example](../examples/bank/) both use SQLite ## License -MIT - see [LICENSE.md](LICENSE.md) +MIT -- see [LICENSE.md](LICENSE.md) diff --git a/eventium-sqlite.cabal b/eventium-sqlite.cabal --- a/eventium-sqlite.cabal +++ b/eventium-sqlite.cabal @@ -1,11 +1,11 @@ cabal-version: 1.12 --- This file has been generated from package.yaml by hpack version 0.37.0. +-- This file has been generated from package.yaml by hpack version 0.38.3. -- -- see: https://github.com/sol/hpack name: eventium-sqlite -version: 0.1.0 +version: 0.2.1 synopsis: SQLite implementations for eventium description: Eventium-sqlite provides a SQLite-based event store implementation for the Eventium event sourcing framework. It uses the Persistent library for type-safe database access and provides efficient event @@ -28,37 +28,55 @@ type: git location: https://github.com/aleks-sidorenko/eventium +flag ci + description: Enable -Werror for CI builds + manual: True + default: False + library exposed-modules: + Eventium.ProjectionCache.Sqlite Eventium.Store.Sqlite other-modules: Paths_eventium_sqlite hs-source-dirs: src - ghc-options: -Wall + default-extensions: + NoFieldSelectors + DuplicateRecordFields + OverloadedRecordDot + ghc-options: -Wall -Wcompat -Widentities -Wincomplete-record-updates -Wredundant-constraints -Wno-type-defaults -Wno-incomplete-uni-patterns build-depends: aeson >=1.5 && <2.3 , base >=4.9 && <5 , bytestring >=0.10 && <0.13 - , eventium-core - , eventium-sql-common + , eventium-core >=0.2.0 && <0.3.0 + , eventium-sql-common >=0.2.0 && <0.3.0 , mtl >=2.2 && <2.4 - , persistent >=2.13 && <2.15 + , persistent >=2.14 && <2.18 , text >=1.2 && <2.2 - , uuid >=1.3 && <1.4 + , uuid ==1.3.* default-language: Haskell2010 + if flag(ci) + ghc-options: -Werror test-suite spec type: exitcode-stdio-1.0 main-is: Spec.hs other-modules: + Eventium.ProjectionCache.SqliteSpec Eventium.Store.SqliteSpec + Eventium.ProjectionCache.Sqlite Eventium.Store.Sqlite Paths_eventium_sqlite hs-source-dirs: tests src - ghc-options: -Wall + default-extensions: + NoFieldSelectors + DuplicateRecordFields + OverloadedRecordDot + ghc-options: -Wall -Wcompat -Widentities -Wincomplete-record-updates -Wredundant-constraints -Wno-type-defaults -Wno-incomplete-uni-patterns build-tool-depends: hspec-discover:hspec-discover build-depends: @@ -66,13 +84,15 @@ , aeson >=1.5 && <2.3 , base >=4.9 && <5 , bytestring >=0.10 && <0.13 - , eventium-core - , eventium-sql-common - , eventium-test-helpers + , eventium-core >=0.2.0 && <0.3.0 + , eventium-sql-common >=0.2.0 && <0.3.0 + , eventium-testkit , hspec , mtl >=2.2 && <2.4 - , persistent >=2.13 && <2.15 + , persistent >=2.14 && <2.18 , persistent-sqlite , text >=1.2 && <2.2 - , uuid >=1.3 && <1.4 + , uuid ==1.3.* default-language: Haskell2010 + if flag(ci) + ghc-options: -Werror diff --git a/src/Eventium/ProjectionCache/Sqlite.hs b/src/Eventium/ProjectionCache/Sqlite.hs new file mode 100644 --- /dev/null +++ b/src/Eventium/ProjectionCache/Sqlite.hs @@ -0,0 +1,52 @@ +module Eventium.ProjectionCache.Sqlite + ( sqliteVersionedProjectionCache, + sqliteGlobalProjectionCache, + sqliteCheckpointStore, + CheckpointName (..), + initializeSqliteProjectionCache, + migrateProjectionSnapshot, + ) +where + +import Control.Monad (void) +import Control.Monad.IO.Class (MonadIO, liftIO) +import Database.Persist.Sql (ConnectionPool, SqlPersistT, runMigrationSilent, runSqlPool) +import Eventium.EventSubscription (CheckpointStore) +import Eventium.ProjectionCache.Sql (CheckpointName (..), ProjectionName, migrateProjectionSnapshot, sqlCheckpointStore, sqlGlobalProjectionCache, sqlVersionedProjectionCache) +import Eventium.ProjectionCache.Types (ProjectionCache) +import Eventium.Store.Class (EventVersion, SequenceNumber) +import Eventium.Store.Sql.JSONString (JSONString) +import Eventium.UUID (UUID) + +-- | SQLite-backed 'ProjectionCache' for per-entity snapshots. +-- Alias for 'sqlVersionedProjectionCache'. +sqliteVersionedProjectionCache :: + (MonadIO m) => + ProjectionName -> + ProjectionCache UUID EventVersion JSONString (SqlPersistT m) +sqliteVersionedProjectionCache = sqlVersionedProjectionCache + +-- | SQLite-backed 'ProjectionCache' for global blob snapshots. +-- Alias for 'sqlGlobalProjectionCache'. +sqliteGlobalProjectionCache :: + (MonadIO m) => + ProjectionName -> + ProjectionCache () SequenceNumber JSONString (SqlPersistT m) +sqliteGlobalProjectionCache = sqlGlobalProjectionCache + +-- | SQLite-backed 'CheckpointStore' for tracking subscription position. +-- Alias for 'sqlCheckpointStore'. +sqliteCheckpointStore :: + (MonadIO m) => + CheckpointName -> + CheckpointStore (SqlPersistT m) SequenceNumber +sqliteCheckpointStore = sqlCheckpointStore + +-- | Run migrations to create the projection_snapshots table in SQLite. +-- Mirrors 'initializeSqliteEventStore' from "Eventium.Store.Sqlite". +initializeSqliteProjectionCache :: + (MonadIO m) => + ConnectionPool -> + m () +initializeSqliteProjectionCache pool = + liftIO $ void $ runSqlPool (runMigrationSilent migrateProjectionSnapshot) pool diff --git a/src/Eventium/Store/Sqlite.hs b/src/Eventium/Store/Sqlite.hs --- a/src/Eventium/Store/Sqlite.hs +++ b/src/Eventium/Store/Sqlite.hs @@ -1,11 +1,11 @@ {-# LANGUAGE OverloadedStrings #-} -{-# LANGUAGE RecordWildCards #-} {-# LANGUAGE TypeFamilies #-} {-# LANGUAGE TypeOperators #-} -- | Defines an Sqlite event store. module Eventium.Store.Sqlite ( sqliteEventStoreWriter, + sqliteEventStoreWriterTagged, initializeSqliteEventStore, module Eventium.Store.Class, module Eventium.Store.Sql, @@ -15,8 +15,6 @@ import Control.Monad.Reader import Data.Text (Text) import Database.Persist -import Database.Persist.Class (SafeToInsert) -import Database.Persist.Names (EntityNameDB (..), FieldNameDB (..)) import Database.Persist.Sql import Eventium.Store.Class import Eventium.Store.Sql @@ -32,6 +30,17 @@ getLatestVersion = sqlMaxEventVersion config maxSqliteVersionSql storeEvents' = sqlStoreEvents config Nothing maxSqliteVersionSql +-- | Like 'sqliteEventStoreWriter' but accepts 'TaggedEvent's, +-- preserving the metadata attached to each event. +sqliteEventStoreWriterTagged :: + (MonadIO m, PersistEntity entity, PersistEntityBackend entity ~ SqlBackend, SafeToInsert entity) => + SqlEventStoreConfig entity serialized -> + VersionedEventStoreWriter (SqlPersistT m) (TaggedEvent serialized) +sqliteEventStoreWriterTagged config = EventStoreWriter $ transactionalExpectedWriteHelper getLatestVersion storeEvents' + where + getLatestVersion = sqlMaxEventVersion config maxSqliteVersionSql + storeEvents' = sqlStoreEventsTagged config Nothing maxSqliteVersionSql + maxSqliteVersionSql :: FieldNameDB -> FieldNameDB -> FieldNameDB -> Text maxSqliteVersionSql (FieldNameDB tableName) (FieldNameDB uuidFieldName) (FieldNameDB versionFieldName) = "SELECT IFNULL(MAX(" <> versionFieldName <> "), -1) FROM " <> tableName <> " WHERE " <> uuidFieldName <> " = ?" @@ -39,17 +48,17 @@ -- | This functions runs the migrations required to create the events table and -- also adds an index on the UUID column. initializeSqliteEventStore :: - (MonadIO m, PersistEntity entity, PersistEntityBackend entity ~ SqlBackend) => + (MonadIO m, PersistEntity entity) => SqlEventStoreConfig entity serialized -> ConnectionPool -> m () -initializeSqliteEventStore SqlEventStoreConfig {..} pool = do +initializeSqliteEventStore config pool = do -- Run migrations _ <- liftIO $ runSqlPool (runMigrationSilent migrateSqlEvent) pool -- Create index on uuid field so retrieval is very fast - let tableName = unEntityNameDB $ tableDBName (sqlEventStoreConfigSequenceMakeEntity undefined undefined undefined) - uuidFieldName = unFieldNameDB $ fieldDBName sqlEventStoreConfigSequenceNumberField + let tableName = unEntityNameDB $ tableDBName (config.sequenceMakeEntity undefined undefined undefined undefined) + uuidFieldName = unFieldNameDB $ fieldDBName config.sequenceNumberField indexSql = "CREATE INDEX IF NOT EXISTS " <> uuidFieldName diff --git a/tests/Eventium/ProjectionCache/SqliteSpec.hs b/tests/Eventium/ProjectionCache/SqliteSpec.hs new file mode 100644 --- /dev/null +++ b/tests/Eventium/ProjectionCache/SqliteSpec.hs @@ -0,0 +1,59 @@ +{-# LANGUAGE OverloadedStrings #-} + +module Eventium.ProjectionCache.SqliteSpec (spec) where + +import Database.Persist.Sqlite +import Eventium.ProjectionCache.Cache (codecProjectionCache) +import Eventium.ProjectionCache.Sqlite +import Eventium.Store.Sqlite +import Eventium.Testkit +import Test.Hspec + +spec :: Spec +spec = do + describe "SQLite versioned projection cache" $ do + versionedProjectionCacheSpec sqliteVersionedProjectionCacheRunner + + describe "SQLite global projection cache" $ do + globalProjectionCacheSpec sqliteGlobalProjectionCacheRunner + + describe "SQLite checkpoint store" $ do + checkpointStoreSpec sqliteCheckpointStoreRunner + +makeStore :: + IO + ( VersionedEventStoreWriter (SqlPersistT IO) CounterEvent, + VersionedEventStoreReader (SqlPersistT IO) CounterEvent, + ConnectionPool + ) +makeStore = do + pool <- liftIO $ runNoLoggingT (createSqlitePool ":memory:" 1) + let writer = codecEventStoreWriter jsonStringCodec $ sqliteEventStoreWriter defaultSqlEventStoreConfig + reader = codecVersionedEventStoreReader jsonStringCodec $ sqlEventStoreReader defaultSqlEventStoreConfig + initializeSqliteEventStore defaultSqlEventStoreConfig pool + initializeSqliteProjectionCache pool + return (writer, reader, pool) + +sqliteVersionedProjectionCacheRunner :: VersionedProjectionCacheRunner (SqlPersistT IO) +sqliteVersionedProjectionCacheRunner = VersionedProjectionCacheRunner $ \action -> do + (writer, reader, pool) <- makeStore + let cache = + codecProjectionCache jsonStringCodec $ + sqliteVersionedProjectionCache (ProjectionName "test_versioned") + runSqlPool (action writer reader cache) pool + +sqliteGlobalProjectionCacheRunner :: GlobalProjectionCacheRunner (SqlPersistT IO) +sqliteGlobalProjectionCacheRunner = GlobalProjectionCacheRunner $ \action -> do + (writer, _, pool) <- makeStore + let globalReader = + codecGlobalEventStoreReader jsonStringCodec $ + sqlGlobalEventStoreReader defaultSqlEventStoreConfig + cache = + codecProjectionCache jsonStringCodec $ + sqliteGlobalProjectionCache (ProjectionName "test_global") + runSqlPool (action writer globalReader cache) pool + +sqliteCheckpointStoreRunner :: CheckpointStoreRunner (SqlPersistT IO) +sqliteCheckpointStoreRunner = CheckpointStoreRunner $ \action -> do + (_, _, pool) <- makeStore + runSqlPool (action (sqliteCheckpointStore (CheckpointName "test_checkpoint"))) pool diff --git a/tests/Eventium/Store/SqliteSpec.hs b/tests/Eventium/Store/SqliteSpec.hs --- a/tests/Eventium/Store/SqliteSpec.hs +++ b/tests/Eventium/Store/SqliteSpec.hs @@ -4,7 +4,7 @@ import Database.Persist.Sqlite import Eventium.Store.Sqlite -import Eventium.TestHelpers +import Eventium.Testkit import Test.Hspec spec :: Spec @@ -22,8 +22,8 @@ makeStore :: IO (VersionedEventStoreWriter (SqlPersistT IO) CounterEvent, VersionedEventStoreReader (SqlPersistT IO) CounterEvent, ConnectionPool) makeStore = do pool <- liftIO $ runNoLoggingT (createSqlitePool ":memory:" 1) - let writer = serializedEventStoreWriter jsonStringSerializer $ sqliteEventStoreWriter defaultSqlEventStoreConfig - reader = serializedVersionedEventStoreReader jsonStringSerializer $ sqlEventStoreReader defaultSqlEventStoreConfig + let writer = codecEventStoreWriter jsonStringCodec $ sqliteEventStoreWriter defaultSqlEventStoreConfig + reader = codecVersionedEventStoreReader jsonStringCodec $ sqlEventStoreReader defaultSqlEventStoreConfig initializeSqliteEventStore defaultSqlEventStoreConfig pool return (writer, reader, pool) @@ -35,7 +35,7 @@ sqliteStoreGlobalRunner :: GlobalStreamEventStoreRunner (SqlPersistT IO) sqliteStoreGlobalRunner = GlobalStreamEventStoreRunner $ \action -> do (writer, _, pool) <- makeStore - let globalStore = serializedGlobalEventStoreReader jsonStringSerializer (sqlGlobalEventStoreReader defaultSqlEventStoreConfig) + let globalStore = codecGlobalEventStoreReader jsonStringCodec (sqlGlobalEventStoreReader defaultSqlEventStoreConfig) runSqlPool (action writer globalStore) pool makeIOStore :: IO (VersionedEventStoreWriter IO CounterEvent, VersionedEventStoreReader IO CounterEvent, ConnectionPool) @@ -53,6 +53,6 @@ sqliteIOStoreGlobalRunner :: GlobalStreamEventStoreRunner IO sqliteIOStoreGlobalRunner = GlobalStreamEventStoreRunner $ \action -> do (writer, _, pool) <- makeIOStore - let globalStore = serializedGlobalEventStoreReader jsonStringSerializer (sqlGlobalEventStoreReader defaultSqlEventStoreConfig) + let globalStore = codecGlobalEventStoreReader jsonStringCodec (sqlGlobalEventStoreReader defaultSqlEventStoreConfig) globalStore' = runEventStoreReaderUsing (`runSqlPool` pool) globalStore action writer globalStore'