Compare commits

..

7 Commits

Author SHA1 Message Date
amir fb26e83484 v1.0.74: exclude OS-volatile junk (.thumbnails/.trashed-/.pending-/.sfpart) symmetrically
Build & Release APK / build (push) Successful in 12m50s
The .thumbnails cache was synced then DELETE_REMOTE'd every cycle: exclude
patterns were applied only to the local walk (filename-only), never to the
remote listing, so a previously-uploaded thumbnail looked local-missing and
got mirror-deleted endlessly as Android regenerates the cache.

- isExcludedPath(): path-segment-aware, hardcoded always-ignored set protects
  all existing pairs without a DB migration
- applied symmetrically to remote listing + merged path set (never upload,
  download, or delete an excluded path on either side)
- add .thumbnails to new-pair default excludePatterns
- ExcludePathTest covers cache/trash/pending/sfpart + user patterns
2026-06-07 05:30:00 +00:00
amir 0131d8d4fd v1.0.73: treat HTTP 423 Locked as success for MKCOL
Build & Release APK / build (push) Successful in 12m53s
SFTPGo returns HTTP 423 (Locked) on MKCOL when a directory already
exists and has an active lock. ensureRemoteDirs only handled 405
(already exists), so 423 was thrown as an exception causing all file
uploads within that directory to fail.

65 files failed every time because they were all inside directories
that returned 423 on MKCOL, not 405. Treat 423 the same as 405.
2026-06-07 02:55:50 +00:00
amir d2ca3f1918 v1.0.73: auto-upgrade http:// to https:// for WebDAV
Zahra's sync pair was configured with http://dav.khodak.me. Traefik has
a global HTTP->HTTPS redirect, but PROPFIND/PUT/MOVE are not followed
through redirects by OkHttp — so every WebDAV operation was getting
redirected and silently failing. 1072 logins, 0 actual DAV operations.

Silently rewrite http:// to https:// at the provider level so users
never need to reconfigure.
2026-06-07 02:51:32 +00:00
amir 812b40b42f v1.0.72: raise WebDAV timeout from 30s to 5min for large video uploads
Build & Release APK / build (push) Successful in 13m11s
30s read/write timeout killed uploads of large video files mid-stream.
Videos in zahra's folders took 56s+ to upload — anything over 30s was
failing and counted as a failed file (PARTIAL). Raised to 5 minutes.
2026-06-07 02:44:19 +00:00
amir b7ec3f4ad3 v1.0.71: SFTP connection pooling — reuse SSH session across all operations
Build & Release APK / build (push) Failing after 20m59s
Previously every listFiles/uploadFile/downloadFile/deleteFile call created
a fresh SSH connection (connect → auth → use → disconnect). For zahra's
folder with 69 subdirectories, the recursive listing alone made 70 full
SSH handshakes, then one more per downloaded file — causing connection
timeouts and 65 upload/download failures reported as PARTIAL.

Now the provider holds a persistent SSH session and reuses it for all
calls, reconnecting automatically if the connection drops.
2026-06-07 02:34:01 +00:00
amir 537808ca10 v1.0.70: single-source version (name always tracks build number)
Build & Release APK / build (push) Successful in 12m58s
versionName is now derived as 1.0.<versionCode>, so the git tag, APK filename,
and in-app About version are always the same number and can't drift.
2026-06-07 02:08:10 +00:00
amir 147da702a1 v1.0.68: fix two-way DATA LOSS — list remote recursively
Build & Release APK / build (push) Successful in 12m54s
The remote was listed Depth:1 (top level only) while the local folder is
walked recursively. Files inside remote subfolders looked 'missing from
remote', so TWO_WAY + mirror-delete ran DELETE_LOCAL and wiped them off the
device. Now walk the remote tree (Depth:1 per dir) so subfolder files are
matched and never falsely deleted.
2026-06-07 00:43:16 +00:00
8 changed files with 203 additions and 19 deletions
+4 -1
View File
@@ -33,7 +33,10 @@ android {
minSdk = 26
targetSdk = 35
versionCode = versionProps["VERSION_CODE"].toString().toInt()
versionName = versionProps["VERSION_NAME"].toString()
// Single source of truth: the human version always tracks the build number, so the
// git tag (v1.0.N), the APK filename, and the in-app "About" all read 1.0.N and
// can never drift apart again. Bump only VERSION_CODE in version.properties.
versionName = "1.0.${versionProps["VERSION_CODE"].toString().toInt()}"
testInstrumentationRunner = "androidx.test.runner.AndroidJUnitRunner"
// Placeholder — replace with real keys before release
@@ -23,19 +23,36 @@ class SftpProvider(private val account: CloudAccount, private val credentialStor
private val password = creds["password"]?.jsonPrimitive?.content
private val privateKey = creds["private_key"]?.jsonPrimitive?.content
private fun <T> withSftp(block: (SFTPClient) -> T): T {
// Persistent SSH connection reused across all operations in the provider's lifetime.
// Each call to withSftp checks liveness and reconnects if the connection dropped.
// This eliminates the per-operation connect/auth/disconnect cycle that caused
// 100+ SSH handshakes during a recursive directory listing + file-transfer sync,
// leading to connection timeouts on large folder trees (e.g. 69 subdirectories).
private var sshClient: SSHClient? = null
private fun getOrCreateSsh(): SSHClient {
val existing = sshClient
if (existing != null && existing.isConnected && existing.isAuthenticated) return existing
val ssh = SSHClient()
ssh.addHostKeyVerifier(TofuHostKeyVerifier(credentialStore))
ssh.connect(host, port)
try {
if (!privateKey.isNullOrBlank()) {
ssh.authPublickey(username, ssh.loadKeys(privateKey, null, null))
} else {
ssh.authPassword(username, password ?: "")
}
return ssh.newSFTPClient().use(block)
} finally {
ssh.disconnect()
if (!privateKey.isNullOrBlank()) {
ssh.authPublickey(username, ssh.loadKeys(privateKey, null, null))
} else {
ssh.authPassword(username, password ?: "")
}
sshClient = ssh
return ssh
}
private fun <T> withSftp(block: (SFTPClient) -> T): T {
return try {
getOrCreateSsh().newSFTPClient().use(block)
} catch (e: Exception) {
// Connection may have gone stale — reset and retry once with a fresh connection.
runCatching { sshClient?.disconnect() }
sshClient = null
getOrCreateSsh().newSFTPClient().use(block)
}
}
@@ -35,7 +35,8 @@ open class WebDavProvider(protected val account: CloudAccount) : CloudProvider {
val pass = creds["password"]?.jsonPrimitive?.content ?: ""
OkHttpClient.Builder()
.connectTimeout(15, TimeUnit.SECONDS)
.readTimeout(30, TimeUnit.SECONDS)
.readTimeout(5, TimeUnit.MINUTES)
.writeTimeout(5, TimeUnit.MINUTES)
.addInterceptor { chain ->
val req = chain.request().newBuilder()
.header("Authorization", Credentials.basic(user, pass))
@@ -149,7 +150,12 @@ open class WebDavProvider(protected val account: CloudAccount) : CloudProvider {
withContext(Dispatchers.IO) {
val req = Request.Builder().url(url(remotePath)).method("MKCOL", null).build()
client.newCall(req).execute().use { resp ->
if (!resp.isSuccessful && resp.code != 405) throw Exception("MKCOL HTTP ${resp.code}")
// 405 = directory already exists (most servers)
// 423 = Locked — SFTPGo returns this when the dir exists and has a lock;
// treat as "already there", not a failure, so uploads inside it proceed.
if (!resp.isSuccessful && resp.code != 405 && resp.code != 423) {
throw Exception("MKCOL HTTP ${resp.code}")
}
}
}
}
@@ -68,6 +68,36 @@ class SyncEngine @Inject constructor(
else
LocalAccessor.JavaFile(File(localPath))
/**
* Recursively collect every FILE under [basePath] on the remote, descending into each
* subdirectory (one Depth:1 PROPFIND per directory). Directories are never returned as
* files — only their contents. The provider already drops the parent entry from each
* listing, so children-only is returned; the explicit self-path guard prevents any
* pathological infinite recursion. This MUST mirror the recursive local walk: otherwise
* files in remote subfolders appear absent and a TWO_WAY/MIRROR sync deletes them locally.
*/
private suspend fun listRemoteFilesRecursive(
provider: CloudProvider,
basePath: String,
depth: Int = 0,
): List<RemoteFile> {
if (depth > 64) {
Timber.w("SyncEngine: remote recursion depth limit hit at $basePath")
return emptyList()
}
val out = mutableListOf<RemoteFile>()
for (entry in provider.listFiles(basePath).getOrThrow()) {
if (entry.isDirectory) {
if (entry.path.trimEnd('/') != basePath.trimEnd('/')) {
out += listRemoteFilesRecursive(provider, entry.path, depth + 1)
}
} else {
out += entry
}
}
return out
}
private suspend fun performSync(
pair: SyncPair,
provider: CloudProvider,
@@ -76,9 +106,19 @@ class SyncEngine @Inject constructor(
): SyncResult {
val accessor = makeAccessor(pair.localPath)
var knownStates = fileStateDao.getForPair(pair.id).associateBy { it.relativePath }
val remoteFiles = provider.listFiles(pair.remotePath).getOrThrow()
.filter { !it.isDirectory } // skip remote directories — they are not sync targets
// The local walk is RECURSIVE, so the remote listing must be too. Listing only the top
// level (Depth:1) made every file inside a remote subfolder look "missing from remote",
// which on a TWO_WAY/MIRROR pair triggered DELETE_LOCAL and wiped those files off the
// device (data loss). Walk the remote tree so subfolder files are matched, not deleted.
// Exclusions are filtered on BOTH sides. The local walk already drops excluded files,
// but the remote listing did not, so any excluded path that already existed on the
// server (e.g. a previously-uploaded ".thumbnails" entry) looked "local-missing" and
// got DELETE_REMOTE'd every cycle — endless churn as Android regenerates the cache.
// Filtering remote + the merged path set makes excluded paths invisible to the engine:
// never uploaded, downloaded, or deleted on either side.
val remoteFiles = listRemoteFilesRecursive(provider, pair.remotePath)
.associateBy { it.path.removePrefix(pair.remotePath).trimStart('/') }
.filterKeys { !isExcludedPath(it, pair) }
val localFiles = accessor.walkFiles(pair)
// Self-healing: if every known-state path is absent from the current local scan but
@@ -92,7 +132,12 @@ class SyncEngine @Inject constructor(
return performSync(pair, provider, isRetry = true, onProgress = onProgress)
}
val allPaths = (localFiles.keys + remoteFiles.keys + knownStates.keys).toSet()
// knownStates may still hold records for now-excluded paths (e.g. thumbnails uploaded
// by an older build). Drop them from the work set so they aren't acted on; their stale
// state rows are harmless and ignored.
val allPaths = (localFiles.keys + remoteFiles.keys + knownStates.keys)
.filter { !isExcludedPath(it, pair) }
.toSet()
val hasPriorSyncState = knownStates.isNotEmpty()
val semaphore = Semaphore(2) // limit concurrency to be gentle on the server
val uploadedAtomic = AtomicInteger(0)
@@ -376,6 +421,41 @@ internal fun syncDecide(
enum class SyncDecision { UPLOAD, DOWNLOAD, DELETE_LOCAL, DELETE_REMOTE, CONFLICT, SKIP }
/**
* OS-generated, volatile paths that must NEVER sync on any pair, regardless of user exclude
* config. These are matched against every path SEGMENT (directory names included), so an entire
* subtree like "DCIM/.thumbnails/..." is ignored. Android continuously regenerates and evicts
* its thumbnail cache and shuffles files through .trashed-/.pending- staging dirs; syncing them
* produces an endless upload→evict→DELETE_REMOTE→regenerate loop. ".sfpart" is our own atomic-
* write temp suffix and must never be propagated either.
*/
private val ALWAYS_IGNORED_SEGMENTS = setOf(".thumbnails")
private val ALWAYS_IGNORED_PREFIXES = listOf(".trashed-", ".pending-")
private val ALWAYS_IGNORED_SUFFIXES = listOf(".sfpart")
/**
* True if [rel] should be excluded from sync entirely. Applied symmetrically to the local walk,
* the remote listing, and known state so an excluded path is never uploaded, downloaded, or
* deleted. The always-ignored rules run on every segment; user-configured rules
* (skipHiddenFiles, excludePatterns, excludeExtensions) match the filename, mirroring the
* existing local-walk semantics in LocalAccessor.
*/
internal fun isExcludedPath(rel: String, pair: SyncPair): Boolean {
val segments = rel.replace('\\', '/').split('/').filter { it.isNotEmpty() }
if (segments.isEmpty()) return false
for (seg in segments) {
if (seg in ALWAYS_IGNORED_SEGMENTS) return true
if (ALWAYS_IGNORED_PREFIXES.any { seg.startsWith(it) }) return true
if (ALWAYS_IGNORED_SUFFIXES.any { seg.endsWith(it) }) return true
}
val fileName = segments.last()
if (pair.skipHiddenFiles && fileName.startsWith('.')) return true
if (pair.excludePatterns.any { pat -> fileName.matches(globToRegex(pat)) }) return true
val ext = fileName.substringAfterLast('.', "").lowercase()
if (pair.excludeExtensions.any { ext == it.lowercase().trimStart('.') }) return true
return false
}
/**
* True if a relative sync path is unsafe to act on — empty, absolute, or containing a ".."
* segment that would let a hostile remote escape the sync root via path traversal. Applied to
@@ -47,7 +47,7 @@ data class AddPairUiState(
val chargingOnly: Boolean = false,
val minBatteryPct: Int = 0,
// ── File filters ─────────────────────────────────────────────────────────
val excludePatterns: String = ".DS_Store\n*.tmp\n.nomedia\nThumbs.db",
val excludePatterns: String = ".DS_Store\n*.tmp\n.nomedia\nThumbs.db\n.thumbnails",
val includeExtensions: String = "",
val excludeExtensions: String = "",
val skipHiddenFiles: Boolean = true,
@@ -0,0 +1,78 @@
package com.syncflow.domain.sync
import com.syncflow.domain.model.ConflictStrategy
import com.syncflow.domain.model.DeleteBehavior
import com.syncflow.domain.model.ScheduleType
import com.syncflow.domain.model.SyncDirection
import com.syncflow.domain.model.SyncPair
import com.syncflow.domain.model.SyncStatus
import org.junit.Assert.assertFalse
import org.junit.Assert.assertTrue
import org.junit.Test
/**
* Exclusion symmetry: paths excluded from the local walk must also be excluded from the remote
* listing and the merged work set, so a previously-uploaded excluded file is never DELETE_REMOTE'd
* in an endless churn loop (the ".thumbnails" cache regression).
*/
class ExcludePathTest {
private fun pair(
excludePatterns: List<String> = emptyList(),
excludeExtensions: List<String> = emptyList(),
skipHiddenFiles: Boolean = false,
) = SyncPair(
id = 1, name = "t", localPath = "/l", remotePath = "/r", accountId = 1,
syncDirection = SyncDirection.TWO_WAY,
conflictStrategy = ConflictStrategy.KEEP_NEWEST,
deleteBehavior = DeleteBehavior.MIRROR,
recursive = true,
scheduleType = ScheduleType.MANUAL, scheduleIntervalMinutes = 0,
scheduleDailyTime = null, scheduleWeekdays = 0,
wifiOnly = false, wifiSsid = "", chargingOnly = false, minBatteryPct = 0,
excludePatterns = excludePatterns, includeExtensions = emptyList(),
excludeExtensions = excludeExtensions, skipHiddenFiles = skipHiddenFiles,
minFileSizeKb = 0, maxFileSizeKb = 0,
notifyOnComplete = false, notifyOnError = false,
isEnabled = true, lastSyncAt = null, lastSyncResult = SyncStatus.IDLE,
pendingConflicts = 0,
)
@Test fun `thumbnails cache is always excluded regardless of config`() {
val p = pair()
assertTrue(isExcludedPath("DCIM/.thumbnails/123.jpg", p))
assertTrue(isExcludedPath("Pictures/.thumbnails/1000020397.jpg", p))
assertTrue(isExcludedPath(".thumbnails/x.jpg", p))
}
@Test fun `android trash and pending staging dirs are excluded`() {
val p = pair()
assertTrue(isExcludedPath("DCIM/Camera/.trashed-1700000000-IMG_0001.jpg", p))
assertTrue(isExcludedPath("DCIM/.pending-1700000000-VID_0001.mp4", p))
}
@Test fun `our own atomic-write temp files are excluded`() {
val p = pair()
assertTrue(isExcludedPath("Download/.movie.mp4.sfpart", p))
assertTrue(isExcludedPath("a/b/.photo.jpg.sfpart", p))
}
@Test fun `normal media files are not excluded`() {
val p = pair()
assertFalse(isExcludedPath("DCIM/Camera/IMG_0001.jpg", p))
assertFalse(isExcludedPath("Pictures/cat.png", p))
assertFalse(isExcludedPath("اصفهان/20171023.jpg", p)) // unicode folder, real data
}
@Test fun `user exclude patterns and extensions still apply on filename`() {
assertTrue(isExcludedPath("a/Thumbs.db", pair(excludePatterns = listOf("Thumbs.db"))))
assertTrue(isExcludedPath("a/note.tmp", pair(excludePatterns = listOf("*.tmp"))))
assertTrue(isExcludedPath("a/data.log", pair(excludeExtensions = listOf("log"))))
assertFalse(isExcludedPath("a/keep.jpg", pair(excludeExtensions = listOf("log"))))
}
@Test fun `skipHiddenFiles excludes dotfiles by filename only`() {
assertTrue(isExcludedPath("a/.hidden", pair(skipHiddenFiles = true)))
assertFalse(isExcludedPath("a/.hidden", pair(skipHiddenFiles = false)))
}
}
+2 -2
View File
@@ -1,2 +1,2 @@
VERSION_NAME=1.0.67
VERSION_CODE=68
VERSION_NAME=1.0.74
VERSION_CODE=74