ECSAM: An Edge-cloud Video Analytics Architecture for the Segment Anything Model

As artificial intelligence continues to evolve, it is increasingly capable of handling a wide range of video analytics tasks with a single large model. One of the key foundation technologies is the Segment Anything Model (SAM), which enables video analytics tasks to be determined on the fly based on user input prompts. However, achieving real-time response in video analytics applications is crucial for user experiences due to the limited communication and computation resources on the edge, especially with SAM, where users may continuously interact by adding or adjusting prompts. In this paper, we propose ECSAM, an edge-cloud computing architecture designed to support SAM for users on the edge. ECSAM integrates new modules on the edge and the cloud to maximize analytics accuracy under both visual and image prompts, while also considering latency constraints. It addresses resource challenges by offering a visual prompt transformation algorithm for visual prompts and efficient workload partitioning for image encoding. ECSAM is implemented on top of the open-source Meta SAM project and develops a case study on a Visual Tour Guide application. Our evaluation indicates that ECSAM significantly enhances the analytics accuracy across various network bandwidths.