New features for 2.5.0

ROADMAP.md

@@ -1,32 +1,81 @@
 # SQLlin Roadmap
 
+Each item has an ID, `Fx`, which stays the same when the item moves to another priority, and under Supported once it is
+done. A new item takes the next number.
+
 ## High Priority
 
-* Support observable queries that return a `Flow` re-emitting whenever the tables they read change, e.g. to invalidate a Paging source
-* Support UNION of SELECTs whose result type isn't the table's row type, such as projections, result columns and joins. `UNION` is typed by the table's row type but reads the rows into its first SELECT's result type, so using the results of such a union fails with a `ClassCastException`
+* F7: Remove the joins written as clauses of SELECT, such as `table SELECT INNER_JOIN<R>(other)`, in the next version after 2.5.0, then let a join start with its first relation, as in `PersonTable INNER_JOIN BookTable`, without `FROM`
 
 ## Medium Priority
 
-* Support WASM platform DSL
-* Support CREATE VIRTUAL TABLE DSL
-* Support CREATE VIEW DSL
-* Support CREATE TRIGGER DSL
-* Support JOIN sub-query DSL
-* Support more functions, such as `coalesce` and `ifnull`, as well as `CAST` and literal values in expressions, e.g. to convert data while copying it to a rebuilt table with `INSERT INTO ... SELECT`
-* Support type converters: store a property of any type through a serializer that encodes it to a type SQLite supports, with type-safe WHERE and SET on its column, e.g. to store instances of kotlinx.datetime
+* F8: Support WASM platform DSL
+* F12: Support type converters: store a property of any type through a serializer that encodes it to a type SQLite supports, with type-safe WHERE and SET on its column, e.g. to store instances of kotlinx.datetime
+* F13: Provide a `PagingSource` for androidx.paging built on observable queries, like Room's `LimitOffsetPagingSource`, so that paging a query doesn't need a hand-written bridge
+* F14: Support constraints in `ALTER_ADD_COLUMN`, which adds a column with its type only: DEFAULT, NOT NULL with DEFAULT, CHECK, COLLATE NOCASE and REFERENCES
+* F15: Support WITHOUT ROWID tables, which save space and a lookup when the primary key isn't an INTEGER
+* F16: Support STRICT tables, which reject values of other types than their columns' (SQLite 3.37.0, Android API 34)
+* F19: Support the collations BINARY and RTRIM besides NOCASE, and COLLATE on the columns of an index and in ORDER BY
+* F21: Support FILTER in aggregate functions (SQLite 3.30.0, Android API 31), and ORDER BY in them, as for `group_concat` (SQLite 3.44.0, Android API 35)
+* F23: Support the other conflict resolutions: INSERT OR ABORT, OR FAIL and OR ROLLBACK, and UPDATE OR IGNORE, OR REPLACE and the others
+* F25: Support ORDER BY and LIMIT in UPDATE and DELETE, which need SQLite's `SQLITE_ENABLE_UPDATE_DELETE_LIMIT` option, which Android's lacks
+* F26: Support INDEXED BY and NOT INDEXED
+* F27: Support choosing how a transaction begins, DEFERRED, IMMEDIATE or EXCLUSIVE, which differs between the platforms now
+* F29: Support EXPLAIN QUERY PLAN, to see which indexes a query uses
+* F30: Support more PRAGMAs, such as integrity_check, quick_check, table_info, index_list, wal_checkpoint, defer_foreign_keys and case_sensitive_like
 
 ## Low Priority
 
-* Support CHECK keyword
-* Support using a query's results within the same transaction, so that a read-modify-write is one transaction
-* Support upsert, `INSERT ... ON CONFLICT (target) DO UPDATE` and `DO NOTHING`, which updates the conflicting row in place where `INSERT OR REPLACE` deletes and re-inserts it
+* F31: Support using a query's results within the same transaction, so that a read-modify-write is one transaction
+* F32: Support upsert, `INSERT ... ON CONFLICT (target) DO UPDATE` and `DO NOTHING`, which updates the conflicting row in place where `INSERT OR REPLACE` deletes and re-inserts it
+* F33: Use a SQLite bundled with SQLlin on Android, such as androidx.sqlite's sqlite-bundled, rather than the system's, which has no FTS5 or R*Tree, and on older Android versions lacks newer SQL such as DROP COLUMN (before API 34) and upsert (before API 30)
+* F34: Support FTS5 and R*Tree virtual tables, which Android's system SQLite lacks
+* F35: Support common table expressions, WITH and WITH RECURSIVE, which hierarchical data such as trees needs, in SELECT and in INSERT, UPDATE and DELETE
+* F36: Support window functions, with OVER, PARTITION BY and WINDOW (SQLite 3.25.0, Android API 30)
+* F37: Support RETURNING in INSERT, UPDATE and DELETE, which returns the rows they write, such as the ids SQLite assigns (SQLite 3.35.0, Android API 34)
+* F38: Support savepoints, SAVEPOINT, RELEASE and ROLLBACK TO, for nested transactions
+* F39: Support ATTACH and DETACH, to query several databases together
+* F40: Support generated columns, `GENERATED ALWAYS AS (...)` (SQLite 3.31.0, Android API 31)
+* F41: Support CREATE TABLE ... AS SELECT
+* F42: Support TEMP tables and views
+* F43: Support custom collations, which the driver registers
+* F44: Support VALUES as a query, and SELECT without FROM, such as `SELECT datetime('now')`
+* F45: Support table-valued functions in FROM, such as json_each, json_tree and pragma_table_info (the JSON ones need SQLite 3.38.0, Android API 34)
+* F47: Support UPDATE ... FROM, which updates rows with the values of another table, and needs the expressions of F9 (SQLite 3.33.0, Android API 34)
+* F48: Support virtual tables other than FTS and R*Tree, such as dbstat
+* F63: Support INSTEAD OF triggers on views, together with INSERT, UPDATE and DELETE on a view, which the DSL doesn't allow, as a view isn't a `Table`, so that such a trigger can be used; left out of the first version of CREATE TRIGGER (F10)
+* F64: Support the JSON functions and JSON's `->` and `->>` operators (SQLite 3.38.0, Android API 34), left out of F46
+* F65: Support the conflict clause of NOT NULL, as `NOT NULL ON CONFLICT REPLACE`, which writes the default value instead of NULL, left out of F17: a property's type declares NOT NULL, so no annotation can hold the clause yet; it could be a new annotation, or a parameter of `@Default`. SQLlin's statements rarely write NULL to a non-null column, so it is of little use for now
 
 ## Supported
 
-* Support INSERT INTO ... SELECT (2.4.0 ✅)
-* Support SQL functions in SELECT results (2.4.0 ✅)
-* Support SELECT projection (2.4.0 ✅)
-* Support INSERT OR IGNORE (2.4.0 ✅)
-* Support INSERT OR REPLACE (2.3.0 ✅)
-* Support FOREIGN KEY DSL (2.2.0 ✅)
-* Support CREATE INDEX DSL (2.2.0 ✅)
+* F17: Support conflict clauses on PRIMARY KEY and UNIQUE constraints (2.5.0 ✅)
+* F18: Support DEFERRABLE INITIALLY DEFERRED foreign keys (2.5.0 ✅)
+* F22: Support NULLS FIRST and NULLS LAST in ORDER BY (2.5.0 ✅)
+* F20: Support ASC and DESC on the columns of an index (2.5.0 ✅)
+* F28: Support ANALYZE, REINDEX and PRAGMA optimize (2.5.0 ✅)
+* F24: Support INSERT ... DEFAULT VALUES (2.5.0 ✅)
+* F10: Support CREATE TRIGGER and DROP TRIGGER, with BEFORE and AFTER triggers on tables (2.5.0 ✅)
+* F9: Support expressions in UPDATE's SET and in INSERT (2.5.0 ✅)
+* F46: Support more expression syntax: CASE, scalar subqueries, COLLATE, bitwise operators and null-safe IS comparisons; REGEXP is left out, as its function differs between the platforms, and the JVM's SQLite has none (2.5.0 ✅)
+* F11: Support more functions, CAST, literal values and arithmetic in expressions (2.5.0 ✅)
+* F6: Support VACUUM and VACUUM INTO (2.5.0 ✅)
+* F5: Support ESCAPE in LIKE (2.5.0 ✅)
+* F3: Support DISTINCT in aggregate functions (2.5.0 ✅)
+* F4: Support partial indexes, `CREATE INDEX ... WHERE` (2.5.0 ✅)
+* F2: Support IF NOT EXISTS and IF EXISTS on CREATE and DROP of tables, views and indexes (2.5.0 ✅)
+* F1: Support DROP INDEX (2.5.0 ✅)
+* F49: Support joins of relations: checked result types, result columns, more than two tables, and RIGHT and FULL OUTER JOIN (2.5.0 ✅)
+* F50: Support subqueries: derived tables in FROM and JOIN, IN and EXISTS (2.5.0 ✅)
+* F51: Support CHECK constraints (2.5.0 ✅)
+* F52: Support CREATE VIRTUAL TABLE for FTS4 and FTS3 full-text search (2.5.0 ✅)
+* F53: Support CREATE VIEW (2.5.0 ✅)
+* F54: Support observable queries that return a `Flow` of their results (2.5.0 ✅)
+* F55: Support compound SELECTs: UNION, UNION ALL, INTERSECT and EXCEPT (2.5.0 ✅)
+* F56: Support INSERT INTO ... SELECT (2.4.0 ✅)
+* F57: Support SQL functions in SELECT results (2.4.0 ✅)
+* F58: Support SELECT projection (2.4.0 ✅)
+* F59: Support INSERT OR IGNORE (2.4.0 ✅)
+* F60: Support INSERT OR REPLACE (2.3.0 ✅)
+* F61: Support FOREIGN KEY DSL (2.2.0 ✅)
+* F62: Support CREATE INDEX DSL (2.2.0 ✅)

gradle.properties

@@ -1,4 +1,4 @@
-VERSION=2.4.0
+VERSION=2.5.0
 GROUP_ID=com.ctrip.kotlin
 
 #Maven Publishing Information

gradle/libs.versions.toml

@@ -1,6 +1,6 @@
 [versions]
 
-kotlin = "2.4.20"
+kotlin = "2.4.21"
 agp = "9.4.1"
 ksp = "2.3.12"
 serialization = "1.11.0"

sample/src/commonMain/kotlin/com/ctrip/sqllin/sample/Sample.kt

@@ -19,7 +19,6 @@ package com.ctrip.sqllin.sample
 import com.ctrip.sqllin.dsl.DSLDBConfiguration
 import com.ctrip.sqllin.dsl.Database
 import com.ctrip.sqllin.dsl.annotation.DBRow
-import com.ctrip.sqllin.dsl.annotation.ExperimentalDSLDatabaseAPI
 import com.ctrip.sqllin.dsl.annotation.PrimaryKey
 import com.ctrip.sqllin.dsl.sql.clause.*
 import com.ctrip.sqllin.dsl.sql.clause.OrderByWay.DESC
@@ -37,7 +36,6 @@ import kotlinx.serialization.Serializable
 
 object Sample {
 
-    @OptIn(ExperimentalDSLDatabaseAPI::class)
     private val db by lazy {
         Database(
             DSLDBConfiguration(

sqllin-driver/build.gradle.kts

@@ -103,8 +103,8 @@ gradle.taskGraph.whenReady {
 }
 
 fun KotlinNativeTarget.setupNativeConfig() {
-    val main by compilations.getting
-    val sqlite3 by main.cinterops.creating {
+    val main = compilations.getByName("main")
+    main.cinterops.create("sqlite3") {
         includeDirs("$projectDir/src/include")
     }
     binaries.all {

sqllin-driver/src/androidHostTest/kotlin/com/ctrip/sqllin/driver/test/AndroidTest.kt

@@ -54,6 +54,9 @@ class AndroidTest {
     @Test
     fun testTransaction() = commonTest.testTransaction()
 
+    @Test
+    fun testFailedCommit() = commonTest.testFailedCommit()
+
     @Test
     fun testConcurrency() = commonTest.testConcurrency()
 

sqllin-driver/src/androidMain/kotlin/com/ctrip/sqllin/driver/AndroidDatabaseConnection.kt

@@ -31,7 +31,19 @@ internal class AndroidDatabaseConnection(private val database: SQLiteDatabase) :
         if (bindParams == null)
             database.execSQL(sql)
         else
-            database.execSQL(sql, bindParams)
+            database.execSQL(sql, Array(bindParams.size) { unsignedAsLong(bindParams[it]) })
+
+    /**
+     * The framework binds a value it doesn't know, such as a `UInt`, as its string, so unsigned values are bound as
+     * the numbers they are, as [bindTypedParameters] does for queries. A `ULong` keeps the bits of a `Long`.
+     */
+    private fun unsignedAsLong(param: Any?): Any? = when (param) {
+        is ULong -> param.toLong()
+        is UInt -> param.toLong()
+        is UShort -> param.toLong()
+        is UByte -> param.toLong()
+        else -> param
+    }
 
     override fun executeInsert(sql: String, bindParams: Array<out Any?>?) = execSQL(sql, bindParams)
 
@@ -85,7 +97,31 @@ internal class AndroidDatabaseConnection(private val database: SQLiteDatabase) :
     }
 
     override fun beginTransaction() = database.beginTransaction()
-    override fun endTransaction() = database.endTransaction()
+    override fun endTransaction() {
+        try {
+            database.endTransaction()
+        } catch (e: Exception) {
+            rollBackOpenTransaction()
+            throw e
+        }
+    }
+
+    /**
+     * Rolls back the transaction that a failed COMMIT leaves open in SQLite, as one does while a deferred foreign key
+     * isn't satisfied: the framework has already let go of it, so that the next one would fail to begin.
+     *
+     * The framework turns a ROLLBACK given to it into its own end of a transaction, which it has none of, so a TEMP
+     * trigger raises the ROLLBACK. The table and the trigger, created in the transaction, are rolled back with it; the
+     * DROP only removes them where SQLite had already rolled back, as it does for some errors.
+     */
+    private fun rollBackOpenTransaction() {
+        runCatching {
+            database.execSQL("CREATE TEMP TABLE IF NOT EXISTS sqllin_rollback(x)")
+            database.execSQL("CREATE TEMP TRIGGER IF NOT EXISTS sqllin_rollback BEFORE INSERT ON sqllin_rollback BEGIN SELECT RAISE(ROLLBACK, 'rollback'); END")
+            database.execSQL("INSERT INTO sqllin_rollback VALUES (1)")
+        }
+        runCatching { database.execSQL("DROP TABLE IF EXISTS temp.sqllin_rollback") }
+    }
     override fun setTransactionSuccessful() = database.setTransactionSuccessful()
 
     override fun close() = database.close()

sqllin-driver/src/commonTest/kotlin/com/ctrip/sqllin/driver/test/CommonBasicTest.kt

@@ -24,6 +24,7 @@ import com.ctrip.sqllin.driver.withTransaction
 import kotlinx.coroutines.*
 import kotlinx.coroutines.test.runTest
 import kotlin.test.assertEquals
+import kotlin.test.assertFails
 
 /**
  * The sqllin-driver common basic test.
@@ -199,6 +200,30 @@ class CommonBasicTest(private val path: DatabasePath) {
         }
     }
 
+    /**
+     * A COMMIT that fails, as one does while a deferred foreign key isn't satisfied, leaves SQLite's transaction open, so
+     * it has to be rolled back: its rows are gone, and the next transaction can begin.
+     */
+    fun testFailedCommit() {
+        openDatabase(getDefaultDBConfig(false)) {
+            it.execSQL("PRAGMA foreign_keys = ON")
+            it.execSQL("CREATE TABLE parent(id INTEGER PRIMARY KEY)")
+            it.execSQL("CREATE TABLE child(parentId INT REFERENCES parent(id) DEFERRABLE INITIALLY DEFERRED)")
+            assertFails {
+                it.withTransaction { connection ->
+                    connection.executeInsert("INSERT INTO child VALUES (?)", arrayOf(1L))
+                }
+            }
+            fun children(): Long = it.withQuery("SELECT count(*) FROM child") { cursor -> cursor.next(); cursor.getLong(0) }
+            assertEquals(0L, children())
+            it.withTransaction { connection ->
+                connection.executeInsert("INSERT INTO child VALUES (?)", arrayOf(1L))
+                connection.executeInsert("INSERT INTO parent VALUES (?)", arrayOf(1L))
+            }
+            assertEquals(1L, children())
+        }
+    }
+
     @OptIn(DelicateCoroutinesApi::class, ExperimentalCoroutinesApi::class)
     fun testConcurrency() = runTest {
         val readWriteConfig = getDefaultDBConfig(false)

sqllin-driver/src/jvmMain/kotlin/com/ctrip/sqllin/driver/AbstractJdbcDatabaseConnection.kt

@@ -56,10 +56,12 @@ internal abstract class AbstractJdbcDatabaseConnection : DatabaseConnection {
                     is Short -> setShort(realIndex, any)
                     is Byte -> setByte(realIndex, any)
 
+                    // As the numbers they are, as the native driver does: setInt(any.toInt()) stored a UInt
+                    // above Int.MAX_VALUE as a negative number. A ULong keeps the bits of a Long.
                     is ULong -> setLong(realIndex, any.toLong())
-                    is UInt -> setInt(realIndex, any.toInt())
-                    is UShort -> setShort(realIndex, any.toShort())
-                    is UByte -> setByte(realIndex, any.toByte())
+                    is UInt -> setLong(realIndex, any.toLong())
+                    is UShort -> setLong(realIndex, any.toLong())
+                    is UByte -> setLong(realIndex, any.toLong())
 
                     is BigDecimal -> setBigDecimal(realIndex, any)
 

sqllin-driver/src/jvmMain/kotlin/com/ctrip/sqllin/driver/JdbcDatabaseConnection.kt

@@ -67,9 +67,16 @@ internal class JdbcDatabaseConnection(private val connection: Connection) : Abst
     }
 
     override fun endTransaction() = try {
-        if (isTransactionSuccess.get())
-            connection.commit()
-        else
+        if (isTransactionSuccess.get()) {
+            try {
+                connection.commit()
+            } catch (e: Exception) {
+                // A COMMIT that fails, as one does while a deferred foreign key isn't satisfied, leaves the
+                // transaction open, so that the statements after it would run in it
+                runCatching { connection.rollback() }
+                throw e
+            }
+        } else
             connection.rollback()
     } finally {
         connection.autoCommit = true

sqllin-driver/src/jvmTest/kotlin/com/ctrip/sqllin/driver/test/JvmTest.kt

@@ -46,6 +46,9 @@ class JvmTest {
     @Test
     fun testTransaction() = commonTest.testTransaction()
 
+    @Test
+    fun testFailedCommit() = commonTest.testFailedCommit()
+
     @Test
     fun testConcurrency() = commonTest.testConcurrency()
 

sqllin-driver/src/nativeMain/kotlin/com/ctrip/sqllin/driver/RealDatabaseConnection.kt

@@ -84,8 +84,18 @@ internal class RealDatabaseConnection(
 
     override fun endTransaction() = transactionLock.withLock {
         try {
-            val sql = if (checkFailTransaction.isSuccessful) "COMMIT;" else "ROLLBACK;"
-            database.rawExecSql(sql)
+            if (checkFailTransaction.isSuccessful) {
+                try {
+                    database.rawExecSql("COMMIT;")
+                } catch (e: Exception) {
+                    // A COMMIT that fails, as one does while a deferred foreign key isn't satisfied, leaves the
+                    // transaction open, so that the statements after it would run in it
+                    runCatching { database.rawExecSql("ROLLBACK;") }
+                    throw e
+                }
+            } else {
+                database.rawExecSql("ROLLBACK;")
+            }
         } finally {
             transaction.value = null
         }

sqllin-driver/src/nativeTest/kotlin/com/ctrip/sqllin/driver/test/NativeTest.kt

@@ -46,6 +46,9 @@ class NativeTest {
     @Test
     fun testTransaction() = commonTest.testTransaction()
 
+    @Test
+    fun testFailedCommit() = commonTest.testFailedCommit()
+
     @Test
     fun testConcurrency() = commonTest.testConcurrency()