Full log
193/335 Test #193: interface.MessagesFormatTest.FormatThreeRoundsPlainText .............................................. Passed 0.01 sec
Start 194: interface.MessagesFormatTest.FormatSystemMessageChatMLStructure
194/335 Test #194: interface.MessagesFormatTest.FormatSystemMessageChatMLStructure ...................................... Passed 0.01 sec
Start 195: interface.MessagesFormatTest.FormatContentWithSpecialCharacters
195/335 Test #195: interface.MessagesFormatTest.FormatContentWithSpecialCharacters ...................................... Passed 0.01 sec
Start 196: interface.MessagesFormatTest.FormatRejectsInvalidMessagesDirectly
196/335 Test #196: interface.MessagesFormatTest.FormatRejectsInvalidMessagesDirectly .................................... Passed 0.07 sec
Start 197: interface.MessagesFormatTest.FormatMessagesNotArrayRejected
197/335 Test #197: interface.MessagesFormatTest.FormatMessagesNotArrayRejected .......................................... Passed 0.01 sec
Start 198: interface.MessagesFormatTest.FormatMultiTurnTwoImagesOrderMatchesPlaceholders
198/335 Test #198: interface.MessagesFormatTest.FormatMultiTurnTwoImagesOrderMatchesPlaceholders ........................ Passed 0.01 sec
Start 199: interface.MessagesFormatTest.FormatMixedTextAndMultipleImagesInSingleTurn
199/335 Test #199: interface.MessagesFormatTest.FormatMixedTextAndMultipleImagesInSingleTurn ............................ Passed 0.01 sec
Start 200: interface.MessagesFormatTest.FormatValidBase64ImagePopulatesBuffers
200/335 Test #200: interface.MessagesFormatTest.FormatValidBase64ImagePopulatesBuffers .................................. Passed 0.01 sec
Start 201: interface.MessagesFormatTest.FormatInvalidBase64DataUrlRejected
201/335 Test #201: interface.MessagesFormatTest.FormatInvalidBase64DataUrlRejected ...................................... Passed 0.07 sec
Start 202: interface.MessagesFormatTest.FormatAssistantReasoningContentInjectedToChatML
202/335 Test #202: interface.MessagesFormatTest.FormatAssistantReasoningContentInjectedToChatML ......................... Passed 0.01 sec
Start 203: interface.MessagesFormatTest.FormatAssistantReasoningOnlyNoContent
203/335 Test #203: interface.MessagesFormatTest.FormatAssistantReasoningOnlyNoContent ................................... Passed 0.02 sec
Start 204: interface.MessagesFormatTest.FormatQwenThinkingDisabledAppendsEmptyBlock
204/335 Test #204: interface.MessagesFormatTest.FormatQwenThinkingDisabledAppendsEmptyBlock ............................. Passed 0.01 sec
Start 205: interface.MessagesFormatTest.FormatQwen3BaseThinkingEnabledNoOpenTag
205/335 Test #205: interface.MessagesFormatTest.FormatQwen3BaseThinkingEnabledNoOpenTag ................................. Passed 0.01 sec
Start 206: interface.MessagesFormatTest.FormatQwen35ThinkingEnabledAppendsOpenTag
206/335 Test #206: interface.MessagesFormatTest.FormatQwen35ThinkingEnabledAppendsOpenTag ............................... Passed 0.01 sec
Start 207: interface.MessagesFormatTest.FormatQwen35ThinkingDisabledAppendsEmptyBlock
207/335 Test #207: interface.MessagesFormatTest.FormatQwen35ThinkingDisabledAppendsEmptyBlock ........................... Passed 0.01 sec
Start 208: interface.MessagesFormatTest.FormatNonQwen3ModelNoThinkingSuffix
208/335 Test #208: interface.MessagesFormatTest.FormatNonQwen3ModelNoThinkingSuffix ..................................... Passed 0.01 sec
Start 209: interface.MessagesFormatTest.FormatQwenEndsWithAssistantNoThinkingSuffix
209/335 Test #209: interface.MessagesFormatTest.FormatQwenEndsWithAssistantNoThinkingSuffix ............................. Passed 0.01 sec
Start 210: interface.MessagesFormatTest.FormatQwenEndsWithAssistantReasoningDoesNotAppendGenerationSuffix
210/335 Test #210: interface.MessagesFormatTest.FormatQwenEndsWithAssistantReasoningDoesNotAppendGenerationSuffix ....... Passed 0.01 sec
Start 211: interface.MessagesFormatTest.ParseThinkingDefaultWhenOmitted
211/335 Test #211: interface.MessagesFormatTest.ParseThinkingDefaultWhenOmitted ......................................... Passed 0.01 sec
Start 212: interface.MessagesFormatTest.ParseThinkingDisabled
212/335 Test #212: interface.MessagesFormatTest.ParseThinkingDisabled ................................................... Passed 0.01 sec
Start 213: interface.MessagesFormatTest.ParseThinkingEnabled
213/335 Test #213: interface.MessagesFormatTest.ParseThinkingEnabled .................................................... Passed 0.01 sec
Start 214: interface.MessagesFormatTest.ParseTopLevelEnableThinkingFalse
214/335 Test #214: interface.MessagesFormatTest.ParseTopLevelEnableThinkingFalse ........................................ Passed 0.01 sec
Start 215: interface.MessagesFormatTest.ParseTopLevelEnableThinkingTrue
215/335 Test #215: interface.MessagesFormatTest.ParseTopLevelEnableThinkingTrue ......................................... Passed 0.01 sec
Start 216: interface.MessagesFormatTest.ParseExtraBodyEnableThinkingAlone
216/335 Test #216: interface.MessagesFormatTest.ParseExtraBodyEnableThinkingAlone ....................................... Passed 0.01 sec
Start 217: interface.MessagesFormatTest.ParseThinkingAllSourcesConsistent
217/335 Test #217: interface.MessagesFormatTest.ParseThinkingAllSourcesConsistent ....................................... Passed 0.01 sec
Start 218: interface.MessagesFormatTest.ParseThinkingAllSourcesConsistentDisabled
218/335 Test #218: interface.MessagesFormatTest.ParseThinkingAllSourcesConsistentDisabled ............................... Passed 0.01 sec
Start 219: interface.MessagesFormatTest.ParseThinkingNonBooleanRejected
219/335 Test #219: interface.MessagesFormatTest.ParseThinkingNonBooleanRejected ......................................... Passed 0.01 sec
Start 220: interface.MessagesFormatTest.ParseThinkingTypeMissingOrNotStringRejected
220/335 Test #220: interface.MessagesFormatTest.ParseThinkingTypeMissingOrNotStringRejected ............................. Passed 0.01 sec
Start 221: interface.MessagesFormatTest.ParseThinkingConflictRejected
221/335 Test #221: interface.MessagesFormatTest.ParseThinkingConflictRejected ........................................... Passed 0.01 sec
Start 222: interface.MessagesFormatTest.ParseReasoningFormatDefaultsToAuto
222/335 Test #222: interface.MessagesFormatTest.ParseReasoningFormatDefaultsToAuto ...................................... Passed 0.01 sec
Start 223: interface.MessagesFormatTest.ParseReasoningFormatExplicitModes
223/335 Test #223: interface.MessagesFormatTest.ParseReasoningFormatExplicitModes ....................................... Passed 0.01 sec
Start 224: interface.MessagesFormatTest.ResolveThinkingStyleFromModelsJson
224/335 Test #224: interface.MessagesFormatTest.ResolveThinkingStyleFromModelsJson ...................................... Passed 0.01 sec
Start 225: interface.MessagesFormatTest.ResolveThinkingStyleQwen3DotXFromJson
225/335 Test #225: interface.MessagesFormatTest.ResolveThinkingStyleQwen3DotXFromJson ................................... Passed 0.01 sec
Start 226: interface.MessagesFormatTest.ResolveThinkingStyleUnregisteredModelReturnsNone
226/335 Test #226: interface.MessagesFormatTest.ResolveThinkingStyleUnregisteredModelReturnsNone ........................ Passed 0.01 sec
Start 227: interface.MessagesFormatTest.ResolveThinkingStyleReturnsNoneForBadConfig
227/335 Test #227: interface.MessagesFormatTest.ResolveThinkingStyleReturnsNoneForBadConfig ............................. Passed 0.07 sec
Start 228: interface.ToolCallRequestTest.ToolsNotArrayRejected
228/335 Test #228: interface.ToolCallRequestTest.ToolsNotArrayRejected .................................................. Passed 0.01 sec
Start 229: interface.ToolCallRequestTest.ToolsObjectRejected
229/335 Test #229: interface.ToolCallRequestTest.ToolsObjectRejected .................................................... Passed 0.01 sec
Start 230: interface.ToolCallRequestTest.ToolChoiceNumberRejected
230/335 Test #230: interface.ToolCallRequestTest.ToolChoiceNumberRejected ............................................... Passed 0.01 sec
Start 231: interface.ToolCallRequestTest.InvalidToolChoiceObjectRejected
231/335 Test #231: interface.ToolCallRequestTest.InvalidToolChoiceObjectRejected ........................................ Passed 0.01 sec
Start 232: interface.ToolCallRequestTest.ToolChoiceArrayRejected
232/335 Test #232: interface.ToolCallRequestTest.ToolChoiceArrayRejected ................................................ Passed 0.01 sec
Start 233: interface.ToolCallForwardTest.ToolsAndToolChoiceForwardsAsToolCallTask
233/335 Test #233: interface.ToolCallForwardTest.ToolsAndToolChoiceForwardsAsToolCallTask ............................... Passed 0.07 sec
Start 234: interface.ToolCallForwardTest.ToolChoiceDefaultsToAutoWhenOmitted
234/335 Test #234: interface.ToolCallForwardTest.ToolChoiceDefaultsToAutoWhenOmitted .................................... Passed 0.07 sec
Start 235: interface.ToolCallForwardTest.ToolChoiceAutoForwardsAsToolCallTask
235/335 Test #235: interface.ToolCallForwardTest.ToolChoiceAutoForwardsAsToolCallTask ................................... Passed 0.07 sec
Start 236: interface.ToolCallForwardTest.ToolChoiceFunctionObjectForwardsAsToolCallTask
236/335 Test #236: interface.ToolCallForwardTest.ToolChoiceFunctionObjectForwardsAsToolCallTask ......................... Passed 0.07 sec
Start 237: interface.ToolCallForwardTest.EmptyToolsArrayDoesNotForwardAsToolCall
237/335 Test #237: interface.ToolCallForwardTest.EmptyToolsArrayDoesNotForwardAsToolCall ................................ Passed 0.07 sec
Start 238: interface.ToolCallForwardTest.MissingToolsDoesNotForwardAsToolCall
238/335 Test #238: interface.ToolCallForwardTest.MissingToolsDoesNotForwardAsToolCall ................................... Passed 0.07 sec
Start 239: interface.ToolCallForwardTest.ToolChoiceWithoutToolsDoesNotForwardAsToolCall
239/335 Test #239: interface.ToolCallForwardTest.ToolChoiceWithoutToolsDoesNotForwardAsToolCall ......................... Passed 0.07 sec
Start 240: api.GewuSmartServeApiTest.NullArgumentsReturnInvalidArgument
240/335 Test #240: api.GewuSmartServeApiTest.NullArgumentsReturnInvalidArgument ......................................... Passed 0.01 sec
Start 241: api.GewuSmartServeApiTest.EmptyArgumentsAreRejectedAtThePublicBoundary
241/335 Test #241: api.GewuSmartServeApiTest.EmptyArgumentsAreRejectedAtThePublicBoundary ............................... Passed 0.01 sec
Start 242: api.GewuSmartServeApiTest.ModelsDirectoryRequiresCanonicalAbsolutePath
242/335 Test #242: api.GewuSmartServeApiTest.ModelsDirectoryRequiresCanonicalAbsolutePath ............................... Passed 0.01 sec
Start 243: api.GewuSmartServeApiTest.ModelsDirectoryRejectsGroupOrWorldWritableRoot
243/335 Test #243: api.GewuSmartServeApiTest.ModelsDirectoryRejectsGroupOrWorldWritableRoot .............................***Skipped 0.01 sec
Start 244: api.GewuSmartServeApiTest.InitializeFinalizeAreIdempotent
244/335 Test #244: api.GewuSmartServeApiTest.InitializeFinalizeAreIdempotent ............................................ Passed 0.61 sec
Start 245: api.GewuSmartServeApiTest.FinalizeOwnsOuterLeaseUntilPublicEntryReturns
245/335 Test #245: api.GewuSmartServeApiTest.FinalizeOwnsOuterLeaseUntilPublicEntryReturns .............................. Passed 0.64 sec
Start 246: api.GewuSmartServeApiTest.FailedFinalizeMustBeRetriedBeforeInitialize
246/335 Test #246: api.GewuSmartServeApiTest.FailedFinalizeMustBeRetriedBeforeInitialize ................................ Passed 0.64 sec
Start 247: api.GewuSmartServeApiTest.FinalizeWaitsForActiveSdkCall
247/335 Test #247: api.GewuSmartServeApiTest.FinalizeWaitsForActiveSdkCall .............................................. Passed 0.65 sec
Start 248: api.GewuSmartServeApiTest.FinalizeIsRejectedFromSdkCallback
248/335 Test #248: api.GewuSmartServeApiTest.FinalizeIsRejectedFromSdkCallback .......................................... Passed 0.64 sec
Start 249: api.GewuSmartServeApiTest.CACertificatePathRoundTrips
249/335 Test #249: api.GewuSmartServeApiTest.CACertificatePathRoundTrips ................................................ Passed 0.04 sec
Start 250: api.GewuSmartServeApiTest.InvalidJsonIsRejectedBeforeServiceInitialization
250/335 Test #250: api.GewuSmartServeApiTest.InvalidJsonIsRejectedBeforeServiceInitialization ........................... Passed 0.02 sec
Start 251: api.GewuSmartServeApiTest.ChatCompletionsStreamInvalidJsonRejected
251/335 Test #251: api.GewuSmartServeApiTest.ChatCompletionsStreamInvalidJsonRejected ................................... Passed 0.02 sec
Start 252: api.GewuSmartServeApiTest.FreeHandlesNullAndValidPointers
252/335 Test #252: api.GewuSmartServeApiTest.FreeHandlesNullAndValidPointers ............................................ Passed 0.61 sec
Start 253: api.GewuSmartServeApiTest.VersionAndListEnginesReturnValidData
253/335 Test #253: api.GewuSmartServeApiTest.VersionAndListEnginesReturnValidData ....................................... Passed 0.65 sec
Start 254: api.GewuSmartServeApiTest.ChatCompletionsRejectsMalformedOrIncompleteRequests
254/335 Test #254: api.GewuSmartServeApiTest.ChatCompletionsRejectsMalformedOrIncompleteRequests ........................ Passed 0.64 sec
Start 255: api.GewuSmartServeApiTest.ConfigJsonValidation
255/335 Test #255: api.GewuSmartServeApiTest.ConfigJsonValidation ....................................................... Passed 0.11 sec
Start 256: api.GewuSmartServeApiTest.ConfigJsonCopiesInputBuffer
256/335 Test #256: api.GewuSmartServeApiTest.ConfigJsonCopiesInputBuffer ................................................ Passed 0.61 sec
Start 257: api.GewuSmartServeApiTest.SetModelsConfigPathLoadsConfigFile
257/335 Test #257: api.GewuSmartServeApiTest.SetModelsConfigPathLoadsConfigFile ......................................... Passed 0.64 sec
Start 258: api.GewuSmartServeApiTest.SetModelsConfigPathRejectsRelativeAndSymlinkPaths
258/335 Test #258: api.GewuSmartServeApiTest.SetModelsConfigPathRejectsRelativeAndSymlinkPaths .......................... Passed 0.04 sec
Start 259: api.GewuSmartServeApiTest.SetModelsConfigJsonRejectsSymlinkedDerivedPathsBeforeCommit
259/335 Test #259: api.GewuSmartServeApiTest.SetModelsConfigJsonRejectsSymlinkedDerivedPathsBeforeCommit ................ Passed 0.62 sec
Start 260: api.GewuSmartServeApiTest.CachedConfigJsonDoesNotReloadRelativePathFromWorkingDirectory
260/335 Test #260: api.GewuSmartServeApiTest.CachedConfigJsonDoesNotReloadRelativePathFromWorkingDirectory .............. Passed 0.64 sec
Start 261: api.GewuSmartServeApiTest.OversizedModelsConfigInputsAreRejectedWithoutReplacingCurrentConfig
261/335 Test #261: api.GewuSmartServeApiTest.OversizedModelsConfigInputsAreRejectedWithoutReplacingCurrentConfig ........ Passed 0.64 sec
Start 262: api.GewuSmartServeApiTest.SetModelsConfigPathRejectsInvalidFilesImmediately
262/335 Test #262: api.GewuSmartServeApiTest.SetModelsConfigPathRejectsInvalidFilesImmediately .......................... Passed 0.11 sec
Start 263: api.GewuSmartServeApiTest.ListModelsUsesGewuOwnerAndOpenAIShape
263/335 Test #263: api.GewuSmartServeApiTest.ListModelsUsesGewuOwnerAndOpenAIShape ...................................... Passed 0.61 sec
Start 264: api.GewuSmartServeApiTest.GetModelInfoAndDownloadInvalidModel
264/335 Test #264: api.GewuSmartServeApiTest.GetModelInfoAndDownloadInvalidModel ........................................ Passed 0.65 sec
Start 265: api.GewuSmartServeApiTest.ConcurrentListModelsAndErrorStringCallsAreThreadSafe
265/335 Test #265: api.GewuSmartServeApiTest.ConcurrentListModelsAndErrorStringCallsAreThreadSafe ....................... Passed 0.64 sec
Start 266: api.GewuSmartServeApiTest.ErrorStringCoversPublicErrorCodes
266/335 Test #266: api.GewuSmartServeApiTest.ErrorStringCoversPublicErrorCodes .......................................... Passed 0.04 sec
Start 267: api.GewuSmartServeApiTest.EightThreadsInferenceStress
267/335 Test #267: api.GewuSmartServeApiTest.EightThreadsInferenceStress ................................................ Passed 0.62 sec
Start 268: api.GewuSmartServeApiTest.RapidCancelRestartStress
268/335 Test #268: api.GewuSmartServeApiTest.RapidCancelRestartStress ................................................... Passed 0.64 sec
Start 269: api.GewuSmartServeApiTest.ConcurrentCancelOfSameRequest
269/335 Test #269: api.GewuSmartServeApiTest.ConcurrentCancelOfSameRequest .............................................. Passed 0.64 sec
Start 270: api.GewuSmartServeApiTest.ConcurrentGetModelInfo
270/335 Test #270: api.GewuSmartServeApiTest.ConcurrentGetModelInfo ..................................................... Passed 0.64 sec
Start 271: api.GewuSmartServeApiTest.ConcurrentDeleteDuringInference
271/335 Test #271: api.GewuSmartServeApiTest.ConcurrentDeleteDuringInference ............................................ Passed 0.65 sec
Start 272: api.GewuSmartServeApiTest.ConcurrentDownloadDuringInference
272/335 Test #272: api.GewuSmartServeApiTest.ConcurrentDownloadDuringInference ..........................................***Failed 0.64 sec
Running main() from /Users/linhan/repos/ci_smartserve/smartserve/third-party/googletest/googletest/src/gtest_main.cc
Note: Google Test filter = GewuSmartServeApiTest.ConcurrentDownloadDuringInference
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from GewuSmartServeApiTest
[ RUN ] GewuSmartServeApiTest.ConcurrentDownloadDuringInference
ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices
ggml_metal_library_init: using embedded metal library
ggml_metal_library_init: loaded in 0.007 sec
ggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)
ggml_metal_device_init: GPU name: MTL0 (Apple M4 Pro)
ggml_metal_device_init: GPU family: MTLGPUFamilyApple9 (1009)
ggml_metal_device_init: GPU family: MTLGPUFamilyCommon3 (3003)
ggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)
ggml_metal_device_init: simdgroup reduction = true
ggml_metal_device_init: simdgroup matrix mul. = true
ggml_metal_device_init: has unified memory = true
ggml_metal_device_init: has bfloat = true
ggml_metal_device_init: has tensor = false
ggml_metal_device_init: use residency sets = true
ggml_metal_device_init: use shared buffers = true
ggml_metal_device_init: recommendedMaxWorkingSetSize = 19069.67 MB
2026-08-19 17:43:36.925 [ERROR][SmartServe][Service] MODEL_CONFIG_PATH is not configured; keeping the previous model state
2026-08-19 17:43:36.925 [ WARN][SmartServe][SmartServeAppInit] No models loaded from config
2026-08-19 17:43:36.925 [ERROR][SmartServe][Service] MODEL_CONFIG_PATH is not configured; keeping the previous model state
2026-08-19 17:43:36.925 [ WARN][SmartServe][Service] concurrent-stress-engine engine not registered, skipping model: cg-download-on-inference
2026-08-19 17:43:36.925 [ WARN][SmartServe][Service] Model file not found, skipping: cg-download-on-inference (/private/var/folders/70/p5lhbxd505j8tnl4nkn1lp800000gn/T/gewu_smartserve_download_on_inference/models/cg-download-on-inference/downloaded.bin)
2026-08-19 17:43:36.925 [ERROR][SmartServe][SmartServeDownloadModel] missing or invalid SHA-256 id=cg-download-on-inference
/Users/linhan/repos/ci_smartserve/smartserve/test/api/gewu_smartserve_concurrent_test.cpp:320: Failure
Expected equality of these values:
GewuSmartServeDownloadModel(modelId.c_str(), nullptr, nullptr)
Which is: -12
GEWU_SMARTSERVE_OK
Which is: 0
[ FAILED ] GewuSmartServeApiTest.ConcurrentDownloadDuringInference (94 ms)
[----------] 1 test from GewuSmartServeApiTest (94 ms total)
[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (94 ms total)
[ PASSED ] 0 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] GewuSmartServeApiTest.ConcurrentDownloadDuringInference
1 FAILED TEST
Start 273: api.GewuSmartServeApiTest.RepeatedDownloadIsIdempotent
273/335 Test #273: api.GewuSmartServeApiTest.RepeatedDownloadIsIdempotent ............................................... Passed 0.64 sec
Start 274: api.GewuSmartServeApiTest.SmartServeEngineWrappersRegisterListAndUnregister
274/335 Test #274: api.GewuSmartServeApiTest.SmartServeEngineWrappersRegisterListAndUnregister .......................... Passed 0.64 sec
Start 275: api.GewuSmartServeApiTest.RegisterEngineCanBeUnregistered
275/335 Test #275: api.GewuSmartServeApiTest.RegisterEngineCanBeUnregistered ............................................ Passed 0.64 sec
Start 276: api.GewuSmartServeApiTest.BuiltInEngineCannotBeExternallyRegisteredOrUnregistered
276/335 Test #276: api.GewuSmartServeApiTest.BuiltInEngineCannotBeExternallyRegisteredOrUnregistered .................... Passed 0.64 sec
Start 277: api.GewuSmartServeApiTest.AppleFoundationModelCanBeRegisteredAfterInitializeAndServeChat
277/335 Test #277: api.GewuSmartServeApiTest.AppleFoundationModelCanBeRegisteredAfterInitializeAndServeChat ............. Passed 0.65 sec
Start 278: api.GewuSmartServeApiTest.SmartServeApiRegistersSystemEngineAndServesChat
278/335 Test #278: api.GewuSmartServeApiTest.SmartServeApiRegistersSystemEngineAndServesChat ............................ Passed 0.65 sec
Start 279: api.GewuSmartServeApiTest.AppleFoundationModelHandlesConcurrentRequests
279/335 Test #279: api.GewuSmartServeApiTest.AppleFoundationModelHandlesConcurrentRequests .............................. Passed 0.65 sec
Start 280: api.GewuSmartServeApiTest.AppleFoundationModelStreamingReturnsChunksAndEof
280/335 Test #280: api.GewuSmartServeApiTest.AppleFoundationModelStreamingReturnsChunksAndEof ........................... Passed 0.65 sec
Start 281: api.GewuSmartServeApiTest.CancelActiveRequestReturnsCancelledOnGetResponse
281/335 Test #281: api.GewuSmartServeApiTest.CancelActiveRequestReturnsCancelledOnGetResponse ........................... Passed 0.65 sec
Start 282: api.GewuSmartServeApiTest.DestroyRequestIsSafeWhileGetResponseIsRunning
282/335 Test #282: api.GewuSmartServeApiTest.DestroyRequestIsSafeWhileGetResponseIsRunning .............................. Passed 0.64 sec
Start 283: api.GewuSmartServeApiTest.FinalizeCancelsActiveRequestAndClearsOpaqueHandle
283/335 Test #283: api.GewuSmartServeApiTest.FinalizeCancelsActiveRequestAndClearsOpaqueHandle .......................... Passed 0.64 sec
Start 284: api.GewuSmartServeApiTest.CancelCompletedRequestReturnsOk
284/335 Test #284: api.GewuSmartServeApiTest.CancelCompletedRequestReturnsOk ............................................ Passed 0.64 sec
Start 285: api.GewuSmartServeApiTest.StreamingRequestCanBeCancelledAfterFirstChunk
285/335 Test #285: api.GewuSmartServeApiTest.StreamingRequestCanBeCancelledAfterFirstChunk .............................. Passed 0.64 sec
Start 286: api.GewuSmartServeApiTest.UnloadReturnsBusyAndDeleteRejectsSystemModelWhileInferenceIsActive
286/335 Test #286: api.GewuSmartServeApiTest.UnloadReturnsBusyAndDeleteRejectsSystemModelWhileInferenceIsActive ......... Passed 0.64 sec
Start 287: api.GewuSmartServeApiTest.DeleteSystemModelDoesNotRemoveSameNamedDirectory
287/335 Test #287: api.GewuSmartServeApiTest.DeleteSystemModelDoesNotRemoveSameNamedDirectory ........................... Passed 0.65 sec
Start 288: api.GewuSmartServeApiTest.InferenceRejectsConfiguredModelWithMissingDeclaredFiles
288/335 Test #288: api.GewuSmartServeApiTest.InferenceRejectsConfiguredModelWithMissingDeclaredFiles .................... Passed 0.64 sec
Start 289: api.GewuSmartServeApiTest.AppleFoundationModelLoadsFromConfigPathAfterInitializeAndRegister
289/335 Test #289: api.GewuSmartServeApiTest.AppleFoundationModelLoadsFromConfigPathAfterInitializeAndRegister .......... Passed 0.65 sec
Start 290: api.GewuSmartServeApiTest.UnregisterBusyEngineThenRetrySucceeds
290/335 Test #290: api.GewuSmartServeApiTest.UnregisterBusyEngineThenRetrySucceeds ...................................... Passed 0.64 sec
Start 291: api.GewuSmartServeApiTest.UnknownModelIdsRejectedByModelApis
291/335 Test #291: api.GewuSmartServeApiTest.UnknownModelIdsRejectedByModelApis ......................................... Passed 0.64 sec
Start 292: api.GewuSmartServeApiTest.ConfiguredModelWithUnsafeStoragePathRejected
292/335 Test #292: api.GewuSmartServeApiTest.ConfiguredModelWithUnsafeStoragePathRejected ............................... Passed 0.65 sec
Start 293: api.GewuSmartServeApiTest.FailInjectionExtractsTraversalModelId
293/335 Test #293: api.GewuSmartServeApiTest.FailInjectionExtractsTraversalModelId ......................................***Failed 0.65 sec
Running main() from /Users/linhan/repos/ci_smartserve/smartserve/third-party/googletest/googletest/src/gtest_main.cc
Note: Google Test filter = GewuSmartServeApiTest.FailInjectionExtractsTraversalModelId
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from GewuSmartServeApiTest
[ RUN ] GewuSmartServeApiTest.FailInjectionExtractsTraversalModelId
ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices
ggml_metal_library_init: using embedded metal library
ggml_metal_library_init: loaded in 0.007 sec
ggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)
ggml_metal_device_init: GPU name: MTL0 (Apple M4 Pro)
ggml_metal_device_init: GPU family: MTLGPUFamilyApple9 (1009)
ggml_metal_device_init: GPU family: MTLGPUFamilyCommon3 (3003)
ggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)
ggml_metal_device_init: simdgroup reduction = true
ggml_metal_device_init: simdgroup matrix mul. = true
ggml_metal_device_init: has unified memory = true
ggml_metal_device_init: has bfloat = true
ggml_metal_device_init: has tensor = false
ggml_metal_device_init: use residency sets = true
ggml_metal_device_init: use shared buffers = true
ggml_metal_device_init: recommendedMaxWorkingSetSize = 19069.67 MB
2026-08-19 17:43:50.475 [ERROR][SmartServe][Service] MODEL_CONFIG_PATH is not configured; keeping the previous model state
2026-08-19 17:43:50.475 [ WARN][SmartServe][SmartServeAppInit] No models loaded from config
2026-08-19 17:43:50.475 [ERROR][SmartServe][Service] MODEL_CONFIG_PATH is not configured; keeping the previous model state
2026-08-19 17:43:50.475 [ERROR][SmartServe][GewuSmartServeChatCompletions] Serve error model=../../evil msg=messages is required and must be a non-empty array
/Users/linhan/repos/ci_smartserve/smartserve/test/api/gewu_smartserve_engine_api_test.cpp:905: Failure
Expected equality of these values:
GewuSmartServeChatCompletions(requestJson.c_str(), &request)
Which is: -12
GEWU_SMARTSERVE_OK
Which is: 0
[ FAILED ] GewuSmartServeApiTest.FailInjectionExtractsTraversalModelId (98 ms)
[----------] 1 test from GewuSmartServeApiTest (98 ms total)
[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (98 ms total)
[ PASSED ] 0 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] GewuSmartServeApiTest.FailInjectionExtractsTraversalModelId
1 FAILED TEST
Start 294: api.GewuSmartServeApiTest.PreinstalledModelWithoutVerifiedDownloadIsNotLoadable
294/335 Test #294: api.GewuSmartServeApiTest.PreinstalledModelWithoutVerifiedDownloadIsNotLoadable ......................***Skipped 0.04 sec
Start 295: api.GewuSmartServeApiTest.MultiFileDownloadReportsInstalledFileProgressProportionally
295/335 Test #295: api.GewuSmartServeApiTest.MultiFileDownloadReportsInstalledFileProgressProportionally ................ Passed 0.62 sec
Start 296: api.GewuSmartServeApiTest.SingleFileDownloadUsesNameInsideModelPathDirectory
296/335 Test #296: api.GewuSmartServeApiTest.SingleFileDownloadUsesNameInsideModelPathDirectory ......................... Passed 0.64 sec
Start 297: api.GewuSmartServeApiTest.DownloadAcceptsCustomHttpsAndRejectsInsecureSources
297/335 Test #297: api.GewuSmartServeApiTest.DownloadAcceptsCustomHttpsAndRejectsInsecureSources ........................ Passed 0.64 sec
Start 298: api.GewuSmartServeApiTest.SingleFileDirectoryPathIsNotReportedDownloaded
298/335 Test #298: api.GewuSmartServeApiTest.SingleFileDirectoryPathIsNotReportedDownloaded ............................. Passed 0.65 sec
Start 299: api.GewuSmartServeApiTest.MultiFileModelRequiresValidChecksumsToBeDownloaded
299/335 Test #299: api.GewuSmartServeApiTest.MultiFileModelRequiresValidChecksumsToBeDownloaded ......................... Passed 0.65 sec
Start 300: api.GewuSmartServeApiTest.DownloadRejectsPathTraversalInModelAndFilePaths
300/335 Test #300: api.GewuSmartServeApiTest.DownloadRejectsPathTraversalInModelAndFilePaths ............................ Passed 0.64 sec
Start 301: api.GewuSmartServeApiTest.ServiceLoadModelsUsesSingleFileNameInsideModelPathDirectory
301/335 Test #301: api.GewuSmartServeApiTest.ServiceLoadModelsUsesSingleFileNameInsideModelPathDirectory ................ Passed 0.64 sec
Start 302: api.GewuSmartServeApiTest.ServiceLoadModelsRejectsDirectoryForSingleFileModel
302/335 Test #302: api.GewuSmartServeApiTest.ServiceLoadModelsRejectsDirectoryForSingleFileModel ........................ Passed 0.64 sec
Start 303: api.GewuSmartServeApiTest.ServiceLoadModelsRejectsEscapedConfiguredPaths
303/335 Test #303: api.GewuSmartServeApiTest.ServiceLoadModelsRejectsEscapedConfiguredPaths ............................. Passed 0.64 sec
Start 304: api.GewuSmartServeApiTest.DownloadPersistsMetadataAndDeleteCleansIt
304/335 Test #304: api.GewuSmartServeApiTest.DownloadPersistsMetadataAndDeleteCleansIt .................................. Passed 0.65 sec
Start 305: api.GewuSmartServeApiTest.PauseDownloadCancelsAndNextDownloadResumes
305/335 Test #305: api.GewuSmartServeApiTest.PauseDownloadCancelsAndNextDownloadResumes ................................. Passed 0.65 sec
Start 306: api.GewuSmartServeApiTest.ProgressCallbackRejectsBlockingLifecycleReentryWithoutDeadlock
306/335 Test #306: api.GewuSmartServeApiTest.ProgressCallbackRejectsBlockingLifecycleReentryWithoutDeadlock ............. Passed 0.65 sec
Start 307: api.GewuSmartServeApiTest.ThreeConcurrentDownloadsOfSameModelAreIdempotent
307/335 Test #307: api.GewuSmartServeApiTest.ThreeConcurrentDownloadsOfSameModelAreIdempotent ........................... Passed 0.64 sec
Start 308: api.GewuSmartServeApiTest.DeleteDuringDownloadCancelsAndRemovesPartialModel
308/335 Test #308: api.GewuSmartServeApiTest.DeleteDuringDownloadCancelsAndRemovesPartialModel .......................... Passed 0.66 sec
Start 309: api.GewuSmartServeApiTest.DownloadStartedDuringDeleteWaitsAndRestoresModel
309/335 Test #309: api.GewuSmartServeApiTest.DownloadStartedDuringDeleteWaitsAndRestoresModel ........................... Passed 0.65 sec
Start 310: api.GewuSmartServeApiTest.InferenceStartedDuringDeleteIsRejected
310/335 Test #310: api.GewuSmartServeApiTest.InferenceStartedDuringDeleteIsRejected ..................................... Passed 0.64 sec
Start 311: api.GewuSmartServeApiTest.ConcurrentRepeatedDeleteHasDeterministicNotFoundResult
311/335 Test #311: api.GewuSmartServeApiTest.ConcurrentRepeatedDeleteHasDeterministicNotFoundResult ..................... Passed 0.64 sec
Start 312: api.GewuSmartServeApiTest.ModelQueriesDoNotWaitForActiveDownload
312/335 Test #312: api.GewuSmartServeApiTest.ModelQueriesDoNotWaitForActiveDownload ..................................... Passed 0.65 sec
Start 313: api.GewuSmartServeApiTest.ModelQueriesReturnBusyDuringDelete
313/335 Test #313: api.GewuSmartServeApiTest.ModelQueriesReturnBusyDuringDelete ......................................... Passed 0.65 sec
Start 314: api.GewuSmartServeApiTest.DeletingLargeModelDoesNotBlockUnrelatedEngineRegistration
314/335 Test #314: api.GewuSmartServeApiTest.DeletingLargeModelDoesNotBlockUnrelatedEngineRegistration .................. Passed 2.00 sec
Start 315: api.GewuSmartServeApiTest.LoadModelsSkipsModelWhileItsFilesAreBeingDeleted
315/335 Test #315: api.GewuSmartServeApiTest.LoadModelsSkipsModelWhileItsFilesAreBeingDeleted ........................... Passed 0.64 sec
Start 316: api.GewuSmartServeApiTest.DeleteUnloadsRuntimeModelWhenFilesAndMetadataAreAlreadyMissing
316/335 Test #316: api.GewuSmartServeApiTest.DeleteUnloadsRuntimeModelWhenFilesAndMetadataAreAlreadyMissing ............. Passed 0.65 sec
Start 317: api.GewuSmartServeApiTest.PreinstalledModelRequiresMatchingChecksum
317/335 Test #317: api.GewuSmartServeApiTest.PreinstalledModelRequiresMatchingChecksum .................................. Passed 0.65 sec
Start 318: api.GewuSmartServeApiTest.PreinstalledModelWithoutChecksumIsRejected
318/335 Test #318: api.GewuSmartServeApiTest.PreinstalledModelWithoutChecksumIsRejected ................................. Passed 0.65 sec
Start 319: api.GewuSmartServeApiTest.DeleteRejectsIntermediateSymlinkEscapingModelRoot
319/335 Test #319: api.GewuSmartServeApiTest.DeleteRejectsIntermediateSymlinkEscapingModelRoot .......................... Passed 0.65 sec
Start 320: functional.GewuSmartServeApiTest.FunctionalLifecycleSmoke
320/335 Test #320: functional.GewuSmartServeApiTest.FunctionalLifecycleSmoke ............................................ Passed 0.65 sec
Start 321: functional.GewuSmartServeApiTest.FunctionalEngineRegistrationSmoke
321/335 Test #321: functional.GewuSmartServeApiTest.FunctionalEngineRegistrationSmoke ................................... Passed 0.64 sec
Start 322: functional.GewuSmartServeApiTest.FunctionalModelConfigSmoke
322/335 Test #322: functional.GewuSmartServeApiTest.FunctionalModelConfigSmoke .......................................... Passed 0.65 sec
Start 323: functional.GewuSmartServeApiTest.FunctionalNonStreamingChatSmoke
323/335 Test #323: functional.GewuSmartServeApiTest.FunctionalNonStreamingChatSmoke ..................................... Passed 0.65 sec
Start 324: functional.GewuSmartServeApiTest.FunctionalStreamingChatSmoke
324/335 Test #324: functional.GewuSmartServeApiTest.FunctionalStreamingChatSmoke ........................................ Passed 0.65 sec
Start 325: functional.GewuSmartServeApiTest.FunctionalCancelRequestSmoke
325/335 Test #325: functional.GewuSmartServeApiTest.FunctionalCancelRequestSmoke ........................................ Passed 0.65 sec
Start 326: functional.GewuSmartServeApiTest.FunctionalSystemModelSmoke
326/335 Test #326: functional.GewuSmartServeApiTest.FunctionalSystemModelSmoke .......................................... Passed 0.65 sec
Start 327: functional.GewuSmartServeApiTest.FunctionalEngineSubmitFailureSmoke
327/335 Test #327: functional.GewuSmartServeApiTest.FunctionalEngineSubmitFailureSmoke .................................. Passed 0.63 sec
Start 328: functional.GewuSmartServeApiTest.FunctionalDownloadCleanupSmoke
328/335 Test #328: functional.GewuSmartServeApiTest.FunctionalDownloadCleanupSmoke ...................................... Passed 0.64 sec
Start 329: functional.GewuSmartServeApiTest.FunctionalDeleteDuringDownloadSmoke
329/335 Test #329: functional.GewuSmartServeApiTest.FunctionalDeleteDuringDownloadSmoke ................................. Passed 0.65 sec
Start 330: model_runtime.MultiSessionTest.MultiSessionAndSingleSessionMultiTask
330/335 Test #330: model_runtime.MultiSessionTest.MultiSessionAndSingleSessionMultiTask .................................***Skipped 0.63 sec
Start 331: model_runtime.MultiSessionTest.LlamaCppHonorsRequestOutputLimit
331/335 Test #331: model_runtime.MultiSessionTest.LlamaCppHonorsRequestOutputLimit ......................................***Skipped 0.64 sec
Start 332: model_runtime.MultiSessionTest.RuntimeCancelThenImmediateNewRequest
332/335 Test #332: model_runtime.MultiSessionTest.RuntimeCancelThenImmediateNewRequest ..................................***Skipped 0.64 sec
Start 333: model_runtime.SmartServeTest.SmartServeLLMInterfaceTest
333/335 Test #333: model_runtime.SmartServeTest.SmartServeLLMInterfaceTest ..............................................***Skipped 0.64 sec
Start 334: model_runtime.SmartServeTest.SmartServeMnnLoraInterfaceTest
334/335 Test #334: model_runtime.SmartServeTest.SmartServeMnnLoraInterfaceTest ..........................................***Skipped 0.65 sec
Start 335: model_runtime.SmartServeTest.SmartServeVLMInterfaceTest
335/335 Test #335: model_runtime.SmartServeTest.SmartServeVLMInterfaceTest ..............................................***Skipped 0.65 sec
99% tests passed, 2 tests failed out of 335
Total Test time (real) = 64.95 sec
The following tests did not run:
243 - api.GewuSmartServeApiTest.ModelsDirectoryRejectsGroupOrWorldWritableRoot (Skipped)
294 - api.GewuSmartServeApiTest.PreinstalledModelWithoutVerifiedDownloadIsNotLoadable (Skipped)
330 - model_runtime.MultiSessionTest.MultiSessionAndSingleSessionMultiTask (Skipped)
331 - model_runtime.MultiSessionTest.LlamaCppHonorsRequestOutputLimit (Skipped)
332 - model_runtime.MultiSessionTest.RuntimeCancelThenImmediateNewRequest (Skipped)
333 - model_runtime.SmartServeTest.SmartServeLLMInterfaceTest (Skipped)
334 - model_runtime.SmartServeTest.SmartServeMnnLoraInterfaceTest (Skipped)
335 - model_runtime.SmartServeTest.SmartServeVLMInterfaceTest (Skipped)
The following tests FAILED:
272 - api.GewuSmartServeApiTest.ConcurrentDownloadDuringInference (Failed)
293 - api.GewuSmartServeApiTest.FailInjectionExtractsTraversalModelId (Failed)
Errors while running CTest
[1;33mSome tests failed; rerunning failed tests once...[0m
Test project /Users/linhan/repos/ci_smartserve/smartserve/build/macos
Start 272: api.GewuSmartServeApiTest.ConcurrentDownloadDuringInference
1/2 Test #272: api.GewuSmartServeApiTest.ConcurrentDownloadDuringInference .......***Failed 0.62 sec
Running main() from /Users/linhan/repos/ci_smartserve/smartserve/third-party/googletest/googletest/src/gtest_main.cc
Note: Google Test filter = GewuSmartServeApiTest.ConcurrentDownloadDuringInference
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from GewuSmartServeApiTest
[ RUN ] GewuSmartServeApiTest.ConcurrentDownloadDuringInference
ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices
ggml_metal_library_init: using embedded metal library
ggml_metal_library_init: loaded in 0.007 sec
ggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)
ggml_metal_device_init: GPU name: MTL0 (Apple M4 Pro)
ggml_metal_device_init: GPU family: MTLGPUFamilyApple9 (1009)
ggml_metal_device_init: GPU family: MTLGPUFamilyCommon3 (3003)
ggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)
ggml_metal_device_init: simdgroup reduction = true
ggml_metal_device_init: simdgroup matrix mul. = true
ggml_metal_device_init: has unified memory = true
ggml_metal_device_init: has bfloat = true
ggml_metal_device_init: has tensor = false
ggml_metal_device_init: use residency sets = true
ggml_metal_device_init: use shared buffers = true
ggml_metal_device_init: recommendedMaxWorkingSetSize = 19069.67 MB
2026-08-19 17:44:19.024 [ERROR][SmartServe][Service] MODEL_CONFIG_PATH is not configured; keeping the previous model state
2026-08-19 17:44:19.024 [ WARN][SmartServe][SmartServeAppInit] No models loaded from config
2026-08-19 17:44:19.024 [ERROR][SmartServe][Service] MODEL_CONFIG_PATH is not configured; keeping the previous model state
2026-08-19 17:44:19.025 [ WARN][SmartServe][Service] concurrent-stress-engine engine not registered, skipping model: cg-download-on-inference
2026-08-19 17:44:19.025 [ WARN][SmartServe][Service] Model file not found, skipping: cg-download-on-inference (/private/var/folders/70/p5lhbxd505j8tnl4nkn1lp800000gn/T/gewu_smartserve_download_on_inference/models/cg-download-on-inference/downloaded.bin)
2026-08-19 17:44:19.025 [ERROR][SmartServe][SmartServeDownloadModel] missing or invalid SHA-256 id=cg-download-on-inference
/Users/linhan/repos/ci_smartserve/smartserve/test/api/gewu_smartserve_concurrent_test.cpp:320: Failure
Expected equality of these values:
GewuSmartServeDownloadModel(modelId.c_str(), nullptr, nullptr)
Which is: -12
GEWU_SMARTSERVE_OK
Which is: 0
[ FAILED ] GewuSmartServeApiTest.ConcurrentDownloadDuringInference (85 ms)
[----------] 1 test from GewuSmartServeApiTest (85 ms total)
[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (85 ms total)
[ PASSED ] 0 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] GewuSmartServeApiTest.ConcurrentDownloadDuringInference
1 FAILED TEST
Start 293: api.GewuSmartServeApiTest.FailInjectionExtractsTraversalModelId
2/2 Test #293: api.GewuSmartServeApiTest.FailInjectionExtractsTraversalModelId ...***Failed 0.64 sec
Running main() from /Users/linhan/repos/ci_smartserve/smartserve/third-party/googletest/googletest/src/gtest_main.cc
Note: Google Test filter = GewuSmartServeApiTest.FailInjectionExtractsTraversalModelId
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from GewuSmartServeApiTest
[ RUN ] GewuSmartServeApiTest.FailInjectionExtractsTraversalModelId
ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices
ggml_metal_library_init: using embedded metal library
ggml_metal_library_init: loaded in 0.007 sec
ggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)
ggml_metal_device_init: GPU name: MTL0 (Apple M4 Pro)
ggml_metal_device_init: GPU family: MTLGPUFamilyApple9 (1009)
ggml_metal_device_init: GPU family: MTLGPUFamilyCommon3 (3003)
ggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)
ggml_metal_device_init: simdgroup reduction = true
ggml_metal_device_init: simdgroup matrix mul. = true
ggml_metal_device_init: has unified memory = true
ggml_metal_device_init: has bfloat = true
ggml_metal_device_init: has tensor = false
ggml_metal_device_init: use residency sets = true
ggml_metal_device_init: use shared buffers = true
ggml_metal_device_init: recommendedMaxWorkingSetSize = 19069.67 MB
2026-08-19 17:44:19.666 [ERROR][SmartServe][Service] MODEL_CONFIG_PATH is not configured; keeping the previous model state
2026-08-19 17:44:19.666 [ WARN][SmartServe][SmartServeAppInit] No models loaded from config
2026-08-19 17:44:19.666 [ERROR][SmartServe][Service] MODEL_CONFIG_PATH is not configured; keeping the previous model state
2026-08-19 17:44:19.666 [ERROR][SmartServe][GewuSmartServeChatCompletions] Serve error model=../../evil msg=messages is required and must be a non-empty array
/Users/linhan/repos/ci_smartserve/smartserve/test/api/gewu_smartserve_engine_api_test.cpp:905: Failure
Expected equality of these values:
GewuSmartServeChatCompletions(requestJson.c_str(), &request)
Which is: -12
GEWU_SMARTSERVE_OK
Which is: 0
[ FAILED ] GewuSmartServeApiTest.FailInjectionExtractsTraversalModelId (94 ms)
[----------] 1 test from GewuSmartServeApiTest (94 ms total)
[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (94 ms total)
[ PASSED ] 0 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] GewuSmartServeApiTest.FailInjectionExtractsTraversalModelId
1 FAILED TEST
0% tests passed, 2 tests failed out of 2
Total Test time (real) = 1.27 sec
The following tests FAILED:
272 - api.GewuSmartServeApiTest.ConcurrentDownloadDuringInference (Failed)
293 - api.GewuSmartServeApiTest.FailInjectionExtractsTraversalModelId (Failed)
Errors while running CTest
=== finished status=failure exit=8 duration=288s timeout_sec=900 attempts=2 ===